-----============= acceptance-small: sanityn ============----- Mon Sep 7 18:24:40 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg303-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 18:25:01 (1788819901) === Starting client oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre2 Mount client oleg303-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre2 Started clients oleg303-client.virtnet: 192.168.203.103@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg303-client.virtnet: executing check_config_client /mnt/lustre oleg303-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg303-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9131c3268800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9131e025f800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9131c3268800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9131e025f800.idle_timeout=debug disable quota as required oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 18:25:35 (1788819935) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265846 s, 39.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7631] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 18:25:37 (1788819937) PASS 0 (9s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 18:25:47 (1788819947) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (6s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 18:25:54 (1788819954) -rw-r--r-- 1 root root 0 Sep 7 18:25 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (5s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 18:25:59 (1788819959) -rw-r--r-- 1 root root 0 Sep 7 18:26 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 18:26:05 (1788819965) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (5s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 18:26:10 (1788819970) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (6s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 18:26:16 (1788819976) total 1 -rwxrwxrwx 1 root root 0 Sep 7 18:25 f2a -rwxrwxrwx 1 root root 0 Sep 7 18:26 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 7 18:25 f2a -rwxrwxrwx 1 root root 0 Sep 7 18:26 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.017864 s, 28.7 kB/s PASS 2e (7s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 18:26:23 (1788819983) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 18:26:26 (1788819986) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.323129 s, 6.5 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 18:26:33 (1788819993) PASS 3 (6s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 18:26:39 (1788819999) PASS 4 (7s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 18:26:46 (1788820006) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 18:26:52 (1788820012) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (6s) == sanityn test 7: remove of open directory on other node ======================================================================= 18:26:58 (1788820018) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 18:27:04 (1788820024) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (5s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 18:27:10 (1788820030) PASS 9a (8s) == sanityn test 9b: append to striped sparse file ======== 18:27:19 (1788820039) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735588 s, 14.3 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0338626 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 18:27:26 (1788820046) 1+0 records in 1+0 records out 1 byte copied, 0.0136668 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0592712 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0132038 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0278826 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0219074 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0204398 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0232027 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0109989 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0220793 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0243532 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0186359 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0105801 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 18:27:33 (1788820053) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0208114 s, 148 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0242309 s, 169 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000317212 s, 9.7 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 18:27:40 (1788820060) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (6s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 18:27:46 (1788820066) warning: '-runtime' deprecated, use '-t runtime' instead - link 352 (time 1788820079.20 total 10.04 last 35.07) - link 662 (time 1788820089.20 total 20.04 last 30.99) - link 949 (time 1788820099.25 total 30.08 last 28.58) - link 1250 (time 1788820109.27 total 40.10 last 30.04) - link 1591 (time 1788820119.28 total 50.12 last 34.05) - link 1856 (time 1788820129.34 total 60.17 last 26.35) - link 2178 (time 1788820139.34 total 70.18 last 32.20) - link 2536 (time 1788820149.36 total 80.20 last 35.73) - link 2797 (time 1788820159.36 total 90.20 last 26.08) - link 3003 (time 1788820169.39 total 100.22 last 20.55) total: 3004 link in 100.22 seconds: 29.97 ops/second using seed 4027012026 running for 100 seconds - stat 5782 (time 1788820081 ; total 11 ; last 11) - stat 10000 (time 1788820088 ; total 18 ; last 7) - stat 15282 (time 1788820099 ; total 29 ; last 11) - stat 20000 (time 1788820108 ; total 38 ; last 9) - stat 26087 (time 1788820119 ; total 49 ; last 11) - stat 29112 (time 1788820130 ; total 60 ; last 11) - stat 30000 (time 1788820132 ; total 62 ; last 2) - stat 32708 (time 1788820143 ; total 73 ; last 11) - stat 35509 (time 1788820154 ; total 84 ; last 11) - stat 37849 (time 1788820165 ; total 95 ; last 11) - stat 40000 (time 1788820168 ; total 98 ; last 3) total: 41836 stats in 100 seconds: 418.359985 stats/second - unlinked 0 (time 1788820120 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3004) error: No such file or directory total: 3003 unlinks in 93 seconds: 32.290321 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21431) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21433) - No such process PASS 12 (155s) == sanityn test 13: test directory page revocation ======= 18:30:21 (1788820221) f13 f13 PASS 13 (10s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 18:30:31 (1788820231) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22618 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (6s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 18:30:37 (1788820237) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23197 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 18:30:45 (1788820245) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23775 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 18:30:52 (1788820252) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24357 Terminated $DIR1/$tdir/sleep 60 PASS 14c (6s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 18:30:58 (1788820258) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24939 Terminated $DIR1/$tdir/sleep 60 PASS 14d (8s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 18:31:06 (1788820266) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (48s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 18:31:54 (1788820314) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.604163 s, 13.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.576578 s, 14.5 MB/s Chance of close/open is 1 in 50 Seed set to 319 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000002[0] 1788820320.414570 mapwrite 0x31b45d thru 0x327b52 (0xc6f6 bytes) 000003[1] 1788820320.508355 punch from 0x0fccfb to 0x109b0e (0xce14 bytes) 000004[1] 1788820320.545400 mapread 0x145300 thru 0x153766 (0xe467 bytes) 000005[0] 1788820320.601086 mapwrite 0x563f15 thru 0x566e1c (0x2f08 bytes) 000006[1] 1788820320.682652 write 0x69f38c thru 0x6aad1f (0xb994 bytes) 000008[1] 1788820320.727375 mapread 0x0a29c0 thru 0x0a3a9d (0x10de bytes) 000009[1] 1788820320.757024 read 0x1d7fb7 thru 0x1dc2b6 (0x4300 bytes) 000013[0] 1788820320.785857 write 0x6f78c0 thru 0x701459 (0x9b9a bytes) 000014[0] 1788820320.806016 punch from 0x061788 to 0x06ab09 (0x9382 bytes) 000015[1] 1788820320.824391 mapread 0x442de6 thru 0x451fd8 (0xf1f3 bytes) 000016[0] 1788820320.950801 read 0x5b392e thru 0x5bd746 (0x9e19 bytes) 000018[0] 1788820320.972090 write 0x0f055d thru 0x0ff9d2 (0xf476 bytes) 000019[1] 1788820321.005986 mapread 0x40b9de thru 0x415913 (0x9f36 bytes) 000023[0] 1788820321.098657 mapread 0x1c3f60 thru 0x1cfadb (0xbb7c bytes) 000026[1] 1788820321.144497 read 0x3bd6cf thru 0x3c3e5a (0x678c bytes) truncating to largest ever: 0x88280d 000028[0] 1788820321.184170 trunc from 0x70145a to 0x88280c (0x1813b3 bytes) 000029[0] 1788820321.286115 mapread 0x332a0a thru 0x33bf12 (0x9509 bytes) 000030[0] 1788820321.353827 write 0x539f3e thru 0x53da5e (0x3b21 bytes) 000031[0] 1788820321.384364 mapwrite 0x779167 thru 0x780e18 (0x7cb2 bytes) 000032[1] 1788820321.573535 mapread 0x45b1bf thru 0x466978 (0xb7ba bytes) 000034[0] 1788820321.655927 mapwrite 0x89230f thru 0x89cf54 (0xac46 bytes) 000036[1] 1788820321.953599 mapread 0x13f73b thru 0x147a2a (0x82f0 bytes) 000037[0] 1788820322.029444 punch from 0x0025e9 to 0x00bc86 (0x969e bytes) 000039[0] 1788820322.103785 trunc from 0x89cf55 to 0x5817c7 (0x31b78d bytes) 000040[0] 1788820322.151662 mapread 0x23ae23 thru 0x240103 (0x52e1 bytes) 000041[0] 1788820322.177508 write 0x341a15 thru 0x34551a (0x3b06 bytes) 000042[1] 1788820322.187087 mapread 0x124dc2 thru 0x126dd1 (0x2010 bytes) 000043[1] 1788820322.261497 write 0x688b80 thru 0x690d98 (0x8219 bytes) 000044[0] 1788820322.312366 write 0x348724 thru 0x3497c4 (0x10a1 bytes) 000045[1] 1788820322.399119 mapread 0x3b3afa thru 0x3c148c (0xd993 bytes) 000046[0] 1788820322.469111 trunc from 0x690d99 to 0x6ceba7 (0x3de0f bytes) 000049[1] 1788820322.583911 punch from 0x4e8c48 to 0x4f170f (0x8ac8 bytes) 000051[1] 1788820322.648883 write 0x51baaf thru 0x52baaa (0xfffc bytes) 000053[1] 1788820322.710268 read 0x1da54a thru 0x1df665 (0x511c bytes) 000054[1] 1788820322.748255 mapread 0x4d10d7 thru 0x4db8b3 (0xa7dd bytes) 000055[1] 1788820322.777210 mapread 0x5150c6 thru 0x51cb2e (0x7a69 bytes) truncating to largest ever: 0x88b324 000056[1] 1788820322.806330 trunc from 0x6ceba8 to 0x88b323 (0x1bc77c bytes) 000058[1] 1788820322.856583 read 0x60d082 thru 0x60ef7f (0x1efe bytes) truncating to largest ever: 0x9a8461 000059[0] 1788820322.901862 trunc from 0x88b324 to 0x9a8460 (0x11d13d bytes) 000060[1] 1788820322.994650 mapwrite 0x7ded6e thru 0x7e7820 (0x8ab3 bytes) 000061[0] 1788820323.271429 read 0x4d6073 thru 0x4de7ba (0x8748 bytes) 000063[0] 1788820323.345224 read 0x8cc326 thru 0x8d8795 (0xc470 bytes) 000064[1] 1788820323.367674 punch from 0x170259 to 0x174322 (0x40ca bytes) 000065[0] 1788820323.389005 trunc from 0x9a8461 to 0x4ec824 (0x4bbc3c bytes) 000068[1] 1788820323.470002 read 0x288635 thru 0x292fe0 (0xa9ac bytes) 000070[0] 1788820323.573057 write 0x9c35ec thru 0x9d0d68 (0xd77d bytes) 000071[1] 1788820323.610798 trunc from 0x9d0d69 to 0x3667e3 (0x66a585 bytes) 000073[0] 1788820323.692117 punch from 0x3351ef to 0x342946 (0xd758 bytes) 000074[1] 1788820323.722439 read 0x2751f0 thru 0x27ab03 (0x5914 bytes) 000075[0] 1788820323.808454 read 0x28c893 thru 0x2968d2 (0xa040 bytes) 000078[0] 1788820323.913155 write 0x82d124 thru 0x83620b (0x90e8 bytes) 000079[0] 1788820323.952350 mapread 0x56f4ba thru 0x576f60 (0x7aa7 bytes) 000080[1] 1788820324.033479 mapread 0x17dee4 thru 0x1883a7 (0xa4c4 bytes) 000081[0] 1788820324.123003 mapread 0x3578f4 thru 0x35a1a3 (0x28b0 bytes) 000084[0] 1788820324.145424 mapread 0x515cd5 thru 0x51994d (0x3c79 bytes) 000085[1] 1788820324.146183 mapread 0x112d9f thru 0x11f0d2 (0xc334 bytes) 000086[1] 1788820324.217471 trunc from 0x83620c to 0x3cfa24 (0x4667e7 bytes) 000087[1] 1788820324.279238 trunc from 0x3cfa25 to 0x191361 (0x23e6c3 bytes) 000089[0] 1788820324.340154 mapread 0x1591a3 thru 0x1663c2 (0xd220 bytes) 000090[0] 1788820324.431682 mapread 0x171f08 thru 0x17553b (0x3634 bytes) 000091[1] 1788820324.464739 read 0x0ab38a thru 0x0b7582 (0xc1f9 bytes) 000093[1] 1788820324.525585 read 0x010f38 thru 0x019628 (0x86f1 bytes) 000095[0] 1788820324.542167 write 0x12a8b6 thru 0x12a9af (0xfa bytes) 000096[0] 1788820324.578225 punch from 0x10cd8e to 0x112ea2 (0x6115 bytes) 000097[0] 1788820324.601990 write 0x6fba29 thru 0x6fbffb (0x5d3 bytes) 000099[0] 1788820324.616486 mapwrite 0x17db96 thru 0x18a215 (0xc680 bytes) 000101[1] 1788820324.774172 read 0x0b9729 thru 0x0c2310 (0x8be8 bytes) 000103[0] 1788820324.814203 trunc from 0x6fbffc to 0x1e3427 (0x518bd4 bytes) 000104[0] 1788820324.851460 punch from 0x05c62a to 0x063604 (0x6fdb bytes) 000105[0] 1788820324.882603 punch from 0x0ab68a to 0x0b1bfc (0x6573 bytes) 000107[0] 1788820324.923385 write 0x21c113 thru 0x21dff8 (0x1ee6 bytes) 000109[0] 1788820324.925882 read 0x159fbd thru 0x165557 (0xb59b bytes) 000111[1] 1788820325.002741 mapread 0x0a5efd thru 0x0a875b (0x285f bytes) 000112[0] 1788820325.056424 mapwrite 0x9703cf thru 0x975fb9 (0x5beb bytes) 000113[1] 1788820325.248696 punch from 0x3b0bda to 0x3bc300 (0xb727 bytes) 000114[0] 1788820325.307200 punch from 0x480718 to 0x48e23e (0xdb27 bytes) 000115[1] 1788820325.330286 mapread 0x093e8e thru 0x093f51 (0xc4 bytes) 000119[1] 1788820325.408482 mapwrite 0x75f7e0 thru 0x76953d (0x9d5e bytes) 000120[1] 1788820325.573794 write 0x43ac41 thru 0x444677 (0x9a37 bytes) 000122[1] 1788820325.589249 punch from 0x5e59b7 to 0x5f1578 (0xbbc2 bytes) 000127[0] 1788820325.720417 trunc from 0x975fba to 0x0ae332 (0x8c7c87 bytes) 000128[0] 1788820325.834021 write 0x9c08ba thru 0x9c09dd (0x124 bytes) 000129[0] 1788820325.869224 write 0x2bd394 thru 0x2cc320 (0xef8d bytes) 000130[0] 1788820325.906744 mapwrite 0x26e707 thru 0x27cb22 (0xe41c bytes) 000132[0] 1788820326.077517 mapwrite 0x6702f6 thru 0x67ce05 (0xcb10 bytes) 000133[1] 1788820326.292051 write 0x9e68b3 thru 0x9f479f (0xdeed bytes) 000134[0] 1788820326.325807 punch from 0x6ecd58 to 0x6faa6c (0xdd15 bytes) 000135[1] 1788820326.363210 trunc from 0x9f47a0 to 0x786f9d (0x26d802 bytes) 000139[1] 1788820326.511768 mapwrite 0x8f2b11 thru 0x8ff9b1 (0xcea1 bytes) 000144[1] 1788820326.790928 punch from 0x4707fb to 0x471ee8 (0x16ee bytes) 000145[1] 1788820326.798574 read 0x241714 thru 0x242f9b (0x1888 bytes) 000148[0] 1788820326.844397 mapread 0x7a7fcf thru 0x7afad7 (0x7b09 bytes) 000149[0] 1788820326.947886 trunc from 0x8ff9b2 to 0x2d52ba (0x62a6f7 bytes) 000150[1] 1788820327.026035 punch from 0x0be6ff to 0x0cb71f (0xd021 bytes) 000153[0] 1788820327.118187 trunc from 0x2d52bb to 0x50d17c (0x237ec2 bytes) 000154[1] 1788820327.159977 punch from 0x07ba79 to 0x08ad44 (0xf2cc bytes) 000155[0] 1788820327.202863 mapwrite 0x5fcd8d thru 0x607b0e (0xad82 bytes) 000159[1] 1788820327.414181 read 0x48f4db thru 0x491719 (0x223f bytes) 000160[0] 1788820327.479757 mapread 0x4a2344 thru 0x4a9c32 (0x78ef bytes) 000163[1] 1788820327.569004 trunc from 0x607b0f to 0x1b1098 (0x456a76 bytes) 000164[0] 1788820327.632497 mapwrite 0x7f8738 thru 0x80188d (0x9156 bytes) 000165[0] 1788820327.882751 mapwrite 0x4d30ca thru 0x4d7406 (0x433d bytes) 000168[0] 1788820327.993159 read 0x3e5454 thru 0x3eb3eb (0x5f98 bytes) 000169[1] 1788820328.033951 write 0x12e6c3 thru 0x12f257 (0xb95 bytes) 000170[0] 1788820328.074401 read 0x115654 thru 0x122367 (0xcd14 bytes) 000172[0] 1788820328.133101 mapread 0x4064ea thru 0x40bf96 (0x5aad bytes) 000173[1] 1788820328.158979 read 0x0260f7 thru 0x02b96d (0x5877 bytes) 000174[1] 1788820328.180305 punch from 0x7a9c68 to 0x7ac449 (0x27e2 bytes) 000175[1] 1788820328.198445 read 0x27bfb1 thru 0x27dcff (0x1d4f bytes) 000178[1] 1788820328.219647 mapread 0x39a05a thru 0x3a826e (0xe215 bytes) 000179[1] 1788820328.252302 punch from 0x013f76 to 0x020659 (0xc6e4 bytes) 000182[0] 1788820328.347736 trunc from 0x80188e to 0x4140cc (0x3ed7c1 bytes) 000183[1] 1788820328.463912 mapwrite 0x465d68 thru 0x46aa93 (0x4d2c bytes) 000186[1] 1788820328.706427 mapread 0x01f2ba thru 0x025867 (0x65ae bytes) 000187[0] 1788820328.778974 read 0x1ba291 thru 0x1bb501 (0x1271 bytes) 000192[1] 1788820328.894908 read 0x17baf5 thru 0x17fe90 (0x439c bytes) 000193[1] 1788820328.935175 read 0x21535c thru 0x21f4cf (0xa174 bytes) 000194[0] 1788820328.963317 mapwrite 0x9866e9 thru 0x989feb (0x3903 bytes) 000197[0] 1788820329.108053 mapread 0x4f165f thru 0x4f937c (0x7d1e bytes) 000198[0] 1788820329.193979 write 0x5b32f5 thru 0x5bb6d4 (0x83e0 bytes) 000199[0] 1788820329.252893 write 0x14b392 thru 0x15620f (0xae7e bytes) 000200[1] 1788820329.277901 write 0x754864 thru 0x75f3f9 (0xab96 bytes) 000201[1] 1788820329.356979 trunc from 0x989fec to 0x7ab193 (0x1dee58 bytes) 000203[1] 1788820329.403542 mapwrite 0x263580 thru 0x27061f (0xd0a0 bytes) 000204[0] 1788820329.489012 trunc from 0x7ab194 to 0x69b8aa (0x10f8e9 bytes) 000206[1] 1788820329.542311 mapread 0x19206e thru 0x192fb1 (0xf44 bytes) 000208[0] 1788820329.604445 mapwrite 0x9a3f13 thru 0x9aefa7 (0xb095 bytes) 000209[0] 1788820329.818381 punch from 0x73a9a2 to 0x73b9a7 (0x1006 bytes) 000210[0] 1788820329.827196 mapwrite 0x95a777 thru 0x965e82 (0xb70c bytes) 000211[0] 1788820329.925049 mapread 0x22485c thru 0x22fa63 (0xb208 bytes) 000213[1] 1788820329.995044 write 0x75edcb thru 0x76ec75 (0xfeab bytes) 000215[0] 1788820330.075076 write 0x7fb4a8 thru 0x80955e (0xe0b7 bytes) 000218[0] 1788820330.154157 read 0x6e4642 thru 0x6efdb1 (0xb770 bytes) 000221[1] 1788820330.169359 mapwrite 0x8e2561 thru 0x8e3abf (0x155f bytes) 000222[0] 1788820330.314287 read 0x030f32 thru 0x03a4c7 (0x9596 bytes) 000223[1] 1788820330.351829 write 0x521e79 thru 0x524bbd (0x2d45 bytes) 000225[0] 1788820330.434667 mapwrite 0x1aa871 thru 0x1aff0f (0x569f bytes) 000226[1] 1788820330.623400 read 0x93ffaa thru 0x94cf21 (0xcf78 bytes) 000228[0] 1788820330.727412 mapwrite 0x99fd70 thru 0x9a67c2 (0x6a53 bytes) 000230[1] 1788820330.836573 trunc from 0x9aefa8 to 0x03975c (0x97584b bytes) 000231[1] 1788820330.898586 mapwrite 0x3fc336 thru 0x400dae (0x4a79 bytes) 000232[1] 1788820331.024535 write 0x642f77 thru 0x64b5c7 (0x8651 bytes) 000233[0] 1788820331.053570 trunc from 0x64b5c8 to 0x7b78d6 (0x16c30f bytes) 000234[0] 1788820331.163393 punch from 0x43dcb2 to 0x440056 (0x23a5 bytes) 000238[0] 1788820331.192251 mapwrite 0x0410a6 thru 0x04bf9f (0xaefa bytes) 000239[1] 1788820331.268421 write 0x3513b3 thru 0x352cce (0x191c bytes) 000241[0] 1788820331.312803 mapwrite 0x171b24 thru 0x1732e6 (0x17c3 bytes) 000242[0] 1788820331.405887 mapwrite 0x39ab85 thru 0x3a4ae1 (0x9f5d bytes) 000243[1] 1788820331.500378 read 0x60f2f2 thru 0x613df9 (0x4b08 bytes) 000244[0] 1788820331.521866 mapwrite 0x4bd514 thru 0x4c3e93 (0x6980 bytes) 000245[1] 1788820331.636469 trunc from 0x7b78d7 to 0x2e87cb (0x4cf10b bytes) 000248[0] 1788820331.740409 punch from 0x2a7946 to 0x2afe06 (0x84c1 bytes) 000249[0] 1788820331.768601 trunc from 0x2e87cc to 0x9957f5 (0x6ad02a bytes) 000250[0] 1788820331.800042 write 0x87e0cc thru 0x88733f (0x9274 bytes) 000251[1] 1788820331.838034 trunc from 0x9957f6 to 0x922f3d (0x728b8 bytes) 000252[0] 1788820331.922002 mapread 0x6ddf65 thru 0x6ebc3c (0xdcd8 bytes) 000254[0] 1788820332.026562 read 0x6f2dbc thru 0x701998 (0xebdd bytes) 000256[1] 1788820332.040617 punch from 0x32b5d3 to 0x332985 (0x73b3 bytes) 000258[1] 1788820332.101384 punch from 0x6a2990 to 0x6a39c0 (0x1031 bytes) 000259[1] 1788820332.137181 mapwrite 0x1d3105 thru 0x1d85e2 (0x54de bytes) truncating to largest ever: 0x9e9390 000260[0] 1788820332.283274 trunc from 0x922f3e to 0x9e938f (0xc6452 bytes) 000261[1] 1788820332.350081 mapwrite 0x695399 thru 0x69b738 (0x63a0 bytes) 000262[1] 1788820332.466470 mapread 0x9a0ef0 thru 0x9a2760 (0x1871 bytes) 000263[1] 1788820332.566415 trunc from 0x9e9390 to 0x91e888 (0xcab07 bytes) 000265[0] 1788820332.608843 write 0x2ec0da thru 0x2f9a3c (0xd963 bytes) 000266[0] 1788820332.652221 trunc from 0x91e889 to 0x47112d (0x4ad75b bytes) 000267[0] 1788820332.741155 punch from 0x242f50 to 0x24c83c (0x98ed bytes) 000270[0] 1788820332.762023 trunc from 0x47112e to 0x91ac13 (0x4a9ae6 bytes) 000272[1] 1788820332.817422 mapread 0x50f50d thru 0x515d0a (0x67fe bytes) 000275[1] 1788820332.893470 punch from 0x7fd110 to 0x80417c (0x706d bytes) 000276[0] 1788820332.978695 mapread 0x49ff43 thru 0x4a161a (0x16d8 bytes) 000278[0] 1788820333.071327 mapread 0x06ac8b thru 0x0766ca (0xba40 bytes) 000279[1] 1788820333.134476 mapread 0x4a0bd5 thru 0x4a957a (0x89a6 bytes) 000280[0] 1788820333.174621 trunc from 0x91ac14 to 0x84c505 (0xce70e bytes) 000283[1] 1788820333.311666 write 0x7d6aaa thru 0x7e516f (0xe6c6 bytes) 000284[1] 1788820333.380398 read 0x211db0 thru 0x219d5c (0x7fad bytes) 000285[0] 1788820333.413255 trunc from 0x84c506 to 0x61b973 (0x230b92 bytes) 000286[0] 1788820333.501368 mapwrite 0x063de7 thru 0x06b34a (0x7564 bytes) 000287[1] 1788820333.571384 mapwrite 0x6c130e thru 0x6cca36 (0xb729 bytes) 000291[0] 1788820333.798326 punch from 0x6004a3 to 0x60755a (0x70b8 bytes) 000292[1] 1788820333.851323 mapwrite 0x95a1ff thru 0x961426 (0x7228 bytes) 000293[1] 1788820334.026701 mapread 0x57ceac thru 0x589cdf (0xce34 bytes) 000294[1] 1788820334.060791 trunc from 0x961427 to 0x7a715c (0x1ba2ca bytes) 000295[1] 1788820334.113488 trunc from 0x7a715d to 0x82ef01 (0x87da5 bytes) 000297[1] 1788820334.171956 write 0x94e569 thru 0x9509ee (0x2486 bytes) 000299[1] 1788820334.195387 mapread 0x65f547 thru 0x66ad6d (0xb827 bytes) 000303[1] 1788820334.271223 write 0x1d5b76 thru 0x1d5fce (0x459 bytes) 000305[1] 1788820334.278683 punch from 0x043569 to 0x049c6b (0x6703 bytes) 000307[1] 1788820334.308562 punch from 0x05a325 to 0x05df45 (0x3c21 bytes) 000308[0] 1788820334.317103 trunc from 0x9509ef to 0x50ed9f (0x441c4f bytes) 000309[1] 1788820334.377430 read 0x3acfc8 thru 0x3b27ca (0x5803 bytes) 000311[0] 1788820334.413972 trunc from 0x50eda0 to 0x6894ef (0x17a750 bytes) 000312[0] 1788820334.460370 read 0x3f5a7c thru 0x3fd80b (0x7d90 bytes) 000315[0] 1788820334.479128 mapwrite 0x7a0ddd thru 0x7aa04d (0x9271 bytes) 000319[1] 1788820334.648357 write 0x1daccc thru 0x1db0ed (0x422 bytes) 000320[1] 1788820334.682639 mapwrite 0x902752 thru 0x90ea7d (0xc32c bytes) 000322[1] 1788820334.824926 mapwrite 0x01a105 thru 0x01d9e3 (0x38df bytes) 000325[0] 1788820334.922792 punch from 0x107f24 to 0x109298 (0x1375 bytes) 000326[1] 1788820334.977952 trunc from 0x90ea7e to 0x8de7aa (0x302d3 bytes) 000328[1] 1788820335.048560 punch from 0x3cbb5f to 0x3ce779 (0x2c1b bytes) 000329[1] 1788820335.080408 mapread 0x74f8ea thru 0x753a58 (0x416f bytes) 000331[0] 1788820335.125001 write 0x114a27 thru 0x11a658 (0x5c32 bytes) 000333[0] 1788820335.190128 trunc from 0x8de7ab to 0x2ed6c7 (0x5f10e3 bytes) 000334[1] 1788820335.230548 mapread 0x0c7a3c thru 0x0d5c04 (0xe1c9 bytes) 000335[1] 1788820335.309441 mapread 0x0e2c45 thru 0x0e5658 (0x2a14 bytes) 000337[1] 1788820335.342284 punch from 0x0cfe08 to 0x0da994 (0xab8d bytes) 000339[0] 1788820335.402131 mapread 0x284adf thru 0x28816c (0x368e bytes) 000341[0] 1788820335.466576 mapread 0x2381aa thru 0x2430c2 (0xaf19 bytes) 000343[1] 1788820335.500058 mapread 0x00f04f thru 0x01ba7a (0xca2c bytes) 000344[0] 1788820335.569445 mapread 0x259048 thru 0x25963e (0x5f7 bytes) 000347[0] 1788820335.656079 trunc from 0x2ed6c8 to 0x96f5c7 (0x681f00 bytes) 000348[1] 1788820335.694647 trunc from 0x96f5c8 to 0x3ac05b (0x5c356c bytes) 000349[1] 1788820335.743466 read 0x3478c3 thru 0x355c13 (0xe351 bytes) 000351[1] 1788820335.857898 mapread 0x24fdf7 thru 0x251a40 (0x1c4a bytes) 000353[0] 1788820335.895236 trunc from 0x3ac05c to 0x48565c (0xd9601 bytes) 000354[0] 1788820335.965300 punch from 0x0e7e2e to 0x0e8a3f (0xc12 bytes) 000355[0] 1788820335.976940 mapwrite 0x0ecf7c thru 0x0f9f65 (0xcfea bytes) 000356[0] 1788820336.072246 mapread 0x2c0b81 thru 0x2c1dfc (0x127c bytes) 000357[0] 1788820336.125423 write 0x34cba2 thru 0x34ebfb (0x205a bytes) 000358[1] 1788820336.186531 mapwrite 0x94f904 thru 0x9593bc (0x9ab9 bytes) 000362[0] 1788820336.479091 mapwrite 0x577d6c thru 0x583379 (0xb60e bytes) 000363[1] 1788820336.643699 write 0x9079ed thru 0x912467 (0xaa7b bytes) 000364[1] 1788820336.674284 trunc from 0x9593bd to 0x4c6ff0 (0x4923cc bytes) 000366[1] 1788820336.795381 mapwrite 0x036a58 thru 0x03f126 (0x86cf bytes) 000368[0] 1788820337.046129 mapread 0x307816 thru 0x31558e (0xdd79 bytes) 000369[1] 1788820337.139642 trunc from 0x4c6ff1 to 0x41d313 (0xa9cdd bytes) 000370[0] 1788820337.173685 trunc from 0x41d314 to 0x55db57 (0x140844 bytes) 000372[0] 1788820337.272437 mapread 0x2cd088 thru 0x2dcafe (0xfa77 bytes) 000377[0] 1788820337.379890 mapwrite 0x252cb1 thru 0x258cd7 (0x6027 bytes) 000379[1] 1788820337.480674 trunc from 0x55db58 to 0x928fee (0x3cb497 bytes) 000380[0] 1788820337.557628 write 0x9645a3 thru 0x973789 (0xf1e7 bytes) 000381[0] 1788820337.614630 write 0x4afb60 thru 0x4baec7 (0xb368 bytes) 000382[1] 1788820337.640572 read 0x4f9768 thru 0x500b43 (0x73dc bytes) 000385[0] 1788820337.761695 mapread 0x7e5c2b thru 0x7ee978 (0x8d4e bytes) 000387[0] 1788820337.825843 write 0x4a6b0f thru 0x4b3f13 (0xd405 bytes) 000388[1] 1788820337.848976 trunc from 0x97378a to 0x2c71cd (0x6ac5bc bytes) 000389[0] 1788820337.911773 trunc from 0x2c71ce to 0x502c31 (0x23ba64 bytes) 000390[0] 1788820337.948563 punch from 0x294430 to 0x29a449 (0x601a bytes) 000391[1] 1788820337.975098 punch from 0x3afca4 to 0x3b099a (0xcf7 bytes) 000392[0] 1788820338.004437 trunc from 0x502c32 to 0x565181 (0x62550 bytes) 000393[0] 1788820338.037650 read 0x45c75f thru 0x46a000 (0xd8a2 bytes) 000394[0] 1788820338.048484 punch from 0x0d632a to 0x0d8b99 (0x2870 bytes) 000395[0] 1788820338.077582 mapread 0x1749c0 thru 0x18041a (0xba5b bytes) 000396[1] 1788820338.162105 mapread 0x0b05e3 thru 0x0b599d (0x53bb bytes) 000397[1] 1788820338.244889 mapwrite 0x51bc5e thru 0x520d0c (0x50af bytes) 000398[0] 1788820338.392841 read 0x385918 thru 0x386c06 (0x12ef bytes) 000399[1] 1788820338.425320 trunc from 0x565182 to 0x7be2f6 (0x259175 bytes) 000400[0] 1788820338.469771 punch from 0x11ec42 to 0x12d855 (0xec14 bytes) 000402[0] 1788820338.547742 mapwrite 0x713ab9 thru 0x719f8d (0x64d5 bytes) 000403[0] 1788820338.693480 punch from 0x3b942b to 0x3c0d7f (0x7955 bytes) 000404[1] 1788820338.700680 mapwrite 0x46bb9e thru 0x46f431 (0x3894 bytes) 000405[1] 1788820338.819752 read 0x02f70e thru 0x039988 (0xa27b bytes) 000406[0] 1788820338.868452 punch from 0x49451d to 0x4a3af3 (0xf5d7 bytes) 000407[0] 1788820338.886338 punch from 0x4ebf02 to 0x4f45f7 (0x86f6 bytes) 000408[1] 1788820338.902953 write 0x87901b thru 0x87b93b (0x2921 bytes) 000409[0] 1788820338.941067 mapwrite 0x3f2ace thru 0x40194e (0xee81 bytes) 000411[0] 1788820339.114292 write 0x9d2015 thru 0x9d39fb (0x19e7 bytes) 000412[1] 1788820339.115697 write 0x9b77a5 thru 0x9b97a9 (0x2005 bytes) 000415[0] 1788820339.199318 read 0x5f9265 thru 0x5fa5fb (0x1397 bytes) 000416[0] 1788820339.221731 mapread 0x4b8fc4 thru 0x4b9a08 (0xa45 bytes) 000418[1] 1788820339.269540 mapread 0x87131c thru 0x8799cd (0x86b2 bytes) 000419[1] 1788820339.345193 mapwrite 0x4b551b thru 0x4c082f (0xb315 bytes) 000422[1] 1788820339.515924 mapwrite 0x5422fc thru 0x546182 (0x3e87 bytes) 000423[1] 1788820339.628937 mapread 0x7b6aef thru 0x7b7ee6 (0x13f8 bytes) 000424[0] 1788820339.694239 write 0x08791e thru 0x08bff3 (0x46d6 bytes) 000427[1] 1788820339.821671 write 0x26236a thru 0x26ac64 (0x88fb bytes) 000430[0] 1788820339.904271 mapread 0x129da6 thru 0x1344f7 (0xa752 bytes) 000432[0] 1788820339.978349 read 0x55f486 thru 0x56e97e (0xf4f9 bytes) 000434[1] 1788820340.016072 mapwrite 0x666a18 thru 0x67180d (0xadf6 bytes) 000437[1] 1788820340.166605 read 0x18e3bf thru 0x19b087 (0xccc9 bytes) 000438[1] 1788820340.220817 mapread 0x3fe81a thru 0x401312 (0x2af9 bytes) 000439[1] 1788820340.280378 punch from 0x94e665 to 0x94f95b (0x12f7 bytes) 000440[0] 1788820340.321360 mapwrite 0x2ef435 thru 0x2fa82a (0xb3f6 bytes) 000442[0] 1788820340.487992 punch from 0x39e146 to 0x3aa0e0 (0xbf9b bytes) 000444[0] 1788820340.565029 punch from 0x68625a to 0x68e5c2 (0x8369 bytes) 000446[0] 1788820340.652059 mapwrite 0x7611f2 thru 0x76fae3 (0xe8f2 bytes) 000447[0] 1788820340.804865 write 0x265203 thru 0x26e9ff (0x97fd bytes) 000448[1] 1788820340.827760 punch from 0x6c023a to 0x6c5bf0 (0x59b7 bytes) 000449[0] 1788820340.903597 trunc from 0x9d39fc to 0x0f36ea (0x8e0311 bytes) 000450[0] 1788820340.947124 trunc from 0x0f36eb to 0x70ce57 (0x61976d bytes) 000451[1] 1788820340.990006 write 0x60c95b thru 0x60d042 (0x6e8 bytes) 000456[0] 1788820341.089516 mapread 0x4c50c3 thru 0x4cd425 (0x8363 bytes) 000458[0] 1788820341.154829 mapwrite 0x3fd14e thru 0x3ff900 (0x27b3 bytes) 000460[0] 1788820341.217613 mapread 0x4a6463 thru 0x4b2b13 (0xc6b1 bytes) 000461[0] 1788820341.253247 write 0x3cc67e thru 0x3ccff8 (0x97b bytes) 000462[1] 1788820341.258940 write 0x6af5a6 thru 0x6bbe63 (0xc8be bytes) 000464[1] 1788820341.322540 mapread 0x2047c2 thru 0x2112ef (0xcb2e bytes) 000466[0] 1788820341.399237 read 0x39db43 thru 0x3a001d (0x24db bytes) 000468[1] 1788820341.426865 read 0x15bc66 thru 0x165d22 (0xa0bd bytes) 000469[1] 1788820341.456208 trunc from 0x70ce58 to 0x5246a5 (0x1e87b2 bytes) 000470[0] 1788820341.517793 mapwrite 0x852603 thru 0x85a1e4 (0x7be2 bytes) 000471[0] 1788820341.636364 mapwrite 0x35b7d4 thru 0x361ec0 (0x66ed bytes) 000473[0] 1788820341.695436 trunc from 0x85a1e5 to 0x98f10a (0x134f26 bytes) 000474[0] 1788820341.736342 write 0x74733b thru 0x75021c (0x8ee2 bytes) 000477[1] 1788820341.779498 punch from 0x8c31ce to 0x8d0e7f (0xdcb2 bytes) 000478[1] 1788820341.810779 mapread 0x155b91 thru 0x160abf (0xaf2f bytes) 000481[1] 1788820341.884431 read 0x327e77 thru 0x333683 (0xb80d bytes) 000482[1] 1788820341.922351 write 0x9ad477 thru 0x9b1530 (0x40ba bytes) 000483[1] 1788820341.947488 read 0x79ba98 thru 0x79e42b (0x2994 bytes) 000485[1] 1788820342.010031 mapwrite 0x884aab thru 0x8858cc (0xe22 bytes) 000486[1] 1788820342.144015 mapwrite 0x16d2e9 thru 0x1744d3 (0x71eb bytes) 000488[1] 1788820342.233691 trunc from 0x9b1531 to 0x43e275 (0x5732bb bytes) 000490[0] 1788820342.355464 trunc from 0x43e276 to 0x101220 (0x33d055 bytes) 000492[0] 1788820342.471247 mapread 0x07d7b8 thru 0x0819ce (0x4217 bytes) 000493[1] 1788820342.512637 punch from 0x013dc9 to 0x02248f (0xe6c7 bytes) 000495[0] 1788820342.578636 trunc from 0x101221 to 0x448d4a (0x347b2a bytes) 000496[0] 1788820342.637990 punch from 0x0862ce to 0x09342b (0xd15e bytes) 000497[1] 1788820342.671213 write 0x29eb54 thru 0x2a6956 (0x7e03 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 342 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[1] 1788820343.439601 mapwrite 0x798000 thru 0x79ce04 (0x4e05 bytes) truncating to largest ever: 0x916d80 000003[1] 1788820343.542764 trunc from 0x79ce05 to 0x916d7f (0x179f7b bytes) 000004[1] 1788820343.567453 mapwrite 0x04e000 thru 0x04e180 (0x181 bytes) 000005[0] 1788820343.667411 punch from 0x4e73ca to 0x4e7992 (0x5c9 bytes) 000006[1] 1788820343.704082 punch from 0x0546a7 to 0x05f79b (0xb0f5 bytes) 000009[1] 1788820343.761892 write_OD 0x3fe000 thru 0x3fefff (0x1000 bytes) 000012[0] 1788820343.847206 read 0x149000 thru 0x155dd3 (0xcdd4 bytes) 000013[0] 1788820343.863965 punch from 0x791cdb to 0x793e00 (0x2126 bytes) 000014[0] 1788820343.900339 mapread 0x02a000 thru 0x02ff32 (0x5f33 bytes) 000016[1] 1788820343.938735 punch from 0x247d6d to 0x25129e (0x9532 bytes) 000017[0] 1788820343.972354 mapread 0x033000 thru 0x0344ab (0x14ac bytes) 000018[0] 1788820343.999256 punch from 0x548b32 to 0x54d526 (0x49f5 bytes) 000019[1] 1788820344.024278 mapread 0x316000 thru 0x31cf41 (0x6f42 bytes) 000023[1] 1788820344.067857 mapwrite 0x04d000 thru 0x05c445 (0xf446 bytes) 000026[0] 1788820344.153865 read 0x55c000 thru 0x55e8a9 (0x28aa bytes) 000027[1] 1788820344.170432 punch from 0x0639fd to 0x0681a9 (0x47ad bytes) 000028[0] 1788820344.204510 read 0x51b000 thru 0x51f4f8 (0x44f9 bytes) 000029[0] 1788820344.237278 mapread 0x379000 thru 0x37a2b5 (0x12b6 bytes) 000030[0] 1788820344.257544 write 0x3b4000 thru 0x3bc2b8 (0x82b9 bytes) 000031[1] 1788820344.300642 trunc from 0x916d80 to 0x673306 (0x2a3a79 bytes) 000032[1] 1788820344.341587 read_OD 0x24f000 thru 0x253fff (0x5000 bytes) 000034[1] 1788820344.366883 mapwrite 0x94b000 thru 0x95a89e (0xf89f bytes) 000035[1] 1788820344.504377 mapread 0x6e5000 thru 0x6ec5eb (0x75ec bytes) 000036[0] 1788820344.531689 mapread 0x75b000 thru 0x766ae2 (0xbae3 bytes) 000037[1] 1788820344.606380 mapread 0x1bd000 thru 0x1c722b (0xa22c bytes) 000040[0] 1788820344.629553 punch from 0x58b0c0 to 0x596427 (0xb368 bytes) 000041[0] 1788820344.665681 write 0x38e000 thru 0x390bad (0x2bae bytes) 000043[1] 1788820344.675844 mapwrite 0x9b5000 thru 0x9b7500 (0x2501 bytes) 000045[0] 1788820344.812542 trunc from 0x9b7501 to 0x87f86a (0x137c96 bytes) 000046[0] 1788820344.882052 trunc from 0x87f86b to 0x6cd5de (0x1b228c bytes) 000047[1] 1788820344.920234 read_OD 0x3fd000 thru 0x401fff (0x5000 bytes) 000048[0] 1788820344.958521 mapread 0x55e000 thru 0x568259 (0xa25a bytes) 000053[0] 1788820344.989970 write 0x7bd000 thru 0x7be1fc (0x11fd bytes) 000056[1] 1788820345.076234 mapread 0x52a000 thru 0x52a02c (0x2d bytes) 000058[1] 1788820345.135158 mapwrite 0x6b3000 thru 0x6b4235 (0x1236 bytes) 000060[0] 1788820345.281015 punch from 0x7ab724 to 0x7bb456 (0xfd33 bytes) 000061[1] 1788820345.297323 read_OD 0x578000 thru 0x579fff (0x2000 bytes) 000062[0] 1788820345.330404 trunc from 0x7be1fd to 0x1e8495 (0x5d5d67 bytes) 000063[1] 1788820345.382408 read_OD 0x12c000 thru 0x12ffff (0x4000 bytes) 000065[0] 1788820345.409008 mapwrite 0x944000 thru 0x94b7c3 (0x77c4 bytes) 000067[0] 1788820345.633248 punch from 0x213543 to 0x21d2b1 (0x9d6f bytes) 000068[1] 1788820345.638353 trunc from 0x94b7c4 to 0x198ef8 (0x7b28cb bytes) 000070[0] 1788820345.715631 mapread 0x054000 thru 0x056348 (0x2349 bytes) 000071[0] 1788820345.770264 mapread 0x084000 thru 0x0933b1 (0xf3b2 bytes) 000077[1] 1788820345.830114 mapwrite 0x969000 thru 0x973f84 (0xaf85 bytes) 000078[0] 1788820346.138416 write 0x75a000 thru 0x7651e3 (0xb1e4 bytes) 000079[0] 1788820346.238263 mapwrite 0x708000 thru 0x7105ea (0x85eb bytes) 000080[0] 1788820346.348623 write 0x05b000 thru 0x062d8f (0x7d90 bytes) 000081[0] 1788820346.368931 write 0x247000 thru 0x24ceaf (0x5eb0 bytes) 000083[0] 1788820346.417516 write 0x54d000 thru 0x55b656 (0xe657 bytes) 000087[1] 1788820346.461444 punch from 0x81c080 to 0x81d793 (0x1714 bytes) 000088[1] 1788820346.506612 trunc from 0x973f85 to 0x8dd9f1 (0x96593 bytes) 000090[1] 1788820346.552449 mapread 0x228000 thru 0x22a1eb (0x21ec bytes) 000091[1] 1788820346.570400 trunc from 0x8dd9f2 to 0x2f4c88 (0x5e8d69 bytes) 000094[1] 1788820346.597576 punch from 0x0d1c98 to 0x0d4513 (0x287c bytes) 000095[0] 1788820346.619949 mapread 0x2d9000 thru 0x2dbfd3 (0x2fd4 bytes) 000097[1] 1788820346.663578 mapwrite 0x6a8000 thru 0x6aa17f (0x2180 bytes) truncating to largest ever: 0x9f63b2 000098[0] 1788820346.860672 trunc from 0x6aa180 to 0x9f63b1 (0x34c232 bytes) 000102[1] 1788820346.968906 punch from 0x3eda36 to 0x3f6390 (0x895b bytes) 000103[0] 1788820347.005318 read 0x13a000 thru 0x13bf01 (0x1f02 bytes) 000105[0] 1788820347.020092 trunc from 0x9f63b2 to 0x849129 (0x1ad288 bytes) 000106[1] 1788820347.072567 trunc from 0x84912a to 0x53d6f5 (0x30ba34 bytes) 000107[1] 1788820347.104701 punch from 0x405c49 to 0x410bc9 (0xaf81 bytes) 000113[0] 1788820347.128410 read 0x537000 thru 0x53b5a4 (0x45a5 bytes) 000115[1] 1788820347.164145 read_OD 0x143000 thru 0x14cfff (0xa000 bytes) 000116[0] 1788820347.190375 trunc from 0x53d6f6 to 0x4b7309 (0x863ec bytes) 000117[0] 1788820347.248627 trunc from 0x4b730a to 0x6a0c02 (0x1e98f9 bytes) 000118[1] 1788820347.272090 mapread 0x097000 thru 0x099f42 (0x2f43 bytes) 000119[1] 1788820347.310508 mapread 0x24d000 thru 0x25b645 (0xe646 bytes) 000120[1] 1788820347.338528 trunc from 0x6a0c03 to 0x3245a6 (0x37c65c bytes) 000122[0] 1788820347.385529 trunc from 0x3245a7 to 0x1a0653 (0x183f53 bytes) 000126[1] 1788820347.469346 read_OD 0x091000 thru 0x091fff (0x1000 bytes) 000127[0] 1788820347.501214 trunc from 0x1a0654 to 0x41cdb3 (0x27c760 bytes) 000129[0] 1788820347.566572 read 0x38a000 thru 0x390ccd (0x6cce bytes) 000130[1] 1788820347.585896 read_OD 0x1c2000 thru 0x1ccfff (0xb000 bytes) 000131[0] 1788820347.626400 read 0x145000 thru 0x1451b4 (0x1b5 bytes) 000132[0] 1788820347.634976 trunc from 0x41cdb4 to 0x59c068 (0x17f2b5 bytes) 000133[0] 1788820347.664295 punch from 0x10c7a8 to 0x1164b5 (0x9d0e bytes) 000134[1] 1788820347.668885 punch from 0x002280 to 0x00edaf (0xcb30 bytes) 000136[1] 1788820347.706232 write_OD 0x539000 thru 0x540fff (0x8000 bytes) 000137[1] 1788820347.782632 write_OD 0x271000 thru 0x273fff (0x3000 bytes) 000139[0] 1788820347.880408 mapwrite 0x936000 thru 0x9448a9 (0xe8aa bytes) 000140[1] 1788820348.034190 write_OD 0x962000 thru 0x96cfff (0xb000 bytes) 000141[1] 1788820348.122558 mapwrite 0x17b000 thru 0x185aea (0xaaeb bytes) 000142[0] 1788820348.280197 mapwrite 0x56a000 thru 0x570da8 (0x6da9 bytes) 000144[0] 1788820348.413855 mapwrite 0x7bc000 thru 0x7cb54e (0xf54f bytes) 000145[0] 1788820348.506356 mapwrite 0x507000 thru 0x50dd75 (0x6d76 bytes) 000148[0] 1788820348.611499 punch from 0x72e64a to 0x72ed25 (0x6dc bytes) 000150[0] 1788820348.630232 mapread 0x418000 thru 0x427932 (0xf933 bytes) 000151[1] 1788820348.653280 mapwrite 0x0a0000 thru 0x0aa13d (0xa13e bytes) 000152[1] 1788820348.826145 mapwrite 0x23b000 thru 0x2482c3 (0xd2c4 bytes) 000155[1] 1788820348.966067 mapread 0x3a4000 thru 0x3ad0ce (0x90cf bytes) 000157[0] 1788820349.067938 mapwrite 0x0d0000 thru 0x0d6fac (0x6fad bytes) 000159[1] 1788820349.219996 mapread 0x33c000 thru 0x346996 (0xa997 bytes) 000160[0] 1788820349.321803 write 0x631000 thru 0x63bd94 (0xad95 bytes) 000162[0] 1788820349.365641 punch from 0x8885dd to 0x89275e (0xa182 bytes) 000163[1] 1788820349.385244 read_OD 0x2ff000 thru 0x30dfff (0xf000 bytes) 000164[1] 1788820349.426207 trunc from 0x96d000 to 0x5c4b03 (0x3a84fc bytes) 000165[0] 1788820349.498523 mapwrite 0x5f5000 thru 0x5ffd5b (0xad5c bytes) 000167[0] 1788820349.678545 punch from 0x506978 to 0x5152e5 (0xe96e bytes) 000169[1] 1788820349.698452 read_OD 0x1b4000 thru 0x1b8fff (0x5000 bytes) 000170[0] 1788820349.726556 mapwrite 0x8e2000 thru 0x8e256c (0x56d bytes) 000171[0] 1788820349.796575 read 0x4a5000 thru 0x4b3cde (0xecdf bytes) 000175[1] 1788820349.845500 mapwrite 0x8e4000 thru 0x8eaa10 (0x6a11 bytes) 000179[1] 1788820350.031086 mapread 0x30a000 thru 0x30fbd9 (0x5bda bytes) 000181[0] 1788820350.065327 trunc from 0x8eaa11 to 0x28f3d6 (0x65b63a bytes) 000185[0] 1788820350.164587 mapwrite 0x293000 thru 0x29c83e (0x983f bytes) 000186[0] 1788820350.302298 read 0x0da000 thru 0x0e0b76 (0x6b77 bytes) 000187[0] 1788820350.373312 punch from 0x184523 to 0x192b38 (0xe616 bytes) 000189[1] 1788820350.393166 trunc from 0x29c83f to 0x7b9545 (0x51cd07 bytes) 000190[1] 1788820350.456581 trunc from 0x7b9546 to 0x929aaa (0x170565 bytes) 000192[1] 1788820350.531601 punch from 0x02bf39 to 0x038094 (0xc15c bytes) 000194[0] 1788820350.541015 write 0x3c6000 thru 0x3d3b51 (0xdb52 bytes) 000198[1] 1788820350.628660 trunc from 0x929aab to 0x09def5 (0x88bbb5 bytes) 000199[1] 1788820350.712898 write_OD 0x836000 thru 0x838fff (0x3000 bytes) 000200[0] 1788820350.803601 punch from 0x695d74 to 0x6a5766 (0xf9f3 bytes) 000201[1] 1788820350.823289 mapread 0x2b7000 thru 0x2c0fd6 (0x9fd7 bytes) 000202[1] 1788820350.869235 punch from 0x227b59 to 0x23317e (0xb626 bytes) 000203[0] 1788820350.917131 punch from 0x4ca573 to 0x4cbf0d (0x199b bytes) 000206[0] 1788820350.949663 trunc from 0x839000 to 0x1d2c2a (0x6663d5 bytes) 000207[1] 1788820351.001192 punch from 0x074daa to 0x078a0d (0x3c64 bytes) 000208[0] 1788820351.018863 read 0x10c000 thru 0x111179 (0x517a bytes) 000210[1] 1788820351.118592 read_OD 0x172000 thru 0x174fff (0x3000 bytes) 000214[0] 1788820351.138624 mapwrite 0x590000 thru 0x5975f3 (0x75f4 bytes) 000215[0] 1788820351.305568 punch from 0x4cf845 to 0x4ded50 (0xf50c bytes) 000217[0] 1788820351.340636 trunc from 0x5975f4 to 0x15a8ba (0x43cd39 bytes) 000220[1] 1788820351.389947 trunc from 0x15a8bb to 0x00b7a8 (0x14f112 bytes) 000221[0] 1788820351.416047 write 0x642000 thru 0x642fda (0xfdb bytes) 000222[1] 1788820351.443583 mapwrite 0x6fa000 thru 0x6fa12b (0x12c bytes) 000224[1] 1788820351.626332 mapwrite 0x539000 thru 0x53cbf0 (0x3bf1 bytes) 000226[0] 1788820351.817509 write 0x355000 thru 0x3611b3 (0xc1b4 bytes) 000227[1] 1788820351.866161 mapwrite 0x044000 thru 0x053e92 (0xfe93 bytes) 000228[0] 1788820352.071638 mapread 0x32c000 thru 0x33675e (0xa75f bytes) 000231[0] 1788820352.208301 read 0x4e4000 thru 0x4f2313 (0xe314 bytes) 000233[1] 1788820352.225825 mapwrite 0x0e3000 thru 0x0e520a (0x220b bytes) 000234[0] 1788820352.371977 punch from 0x40ef13 to 0x414ad1 (0x5bbf bytes) 000237[1] 1788820352.415118 trunc from 0x6fa12c to 0x370386 (0x389da5 bytes) 000239[0] 1788820352.453003 read 0x1f2000 thru 0x1fb79f (0x97a0 bytes) 000241[0] 1788820352.481619 punch from 0x1aa440 to 0x1ab4a7 (0x1068 bytes) 000242[0] 1788820352.498067 mapwrite 0x23d000 thru 0x247ee8 (0xaee9 bytes) 000243[0] 1788820352.635813 mapwrite 0x356000 thru 0x35713e (0x113f bytes) 000246[1] 1788820352.708242 mapread 0x1de000 thru 0x1e73fb (0x93fc bytes) 000247[1] 1788820352.831649 read_OD 0x1fb000 thru 0x203fff (0x9000 bytes) 000248[0] 1788820352.842976 write 0x7b6000 thru 0x7c3957 (0xd958 bytes) 000250[1] 1788820352.897269 punch from 0x73e7e2 to 0x744f96 (0x67b5 bytes) 000251[0] 1788820352.964862 punch from 0x417a75 to 0x4247db (0xcd67 bytes) 000252[1] 1788820352.984670 mapread 0x009000 thru 0x01264b (0x964c bytes) 000254[0] 1788820353.088223 mapwrite 0x3a6000 thru 0x3a7430 (0x1431 bytes) 000257[0] 1788820353.226004 punch from 0x569de8 to 0x56a9ad (0xbc6 bytes) 000258[1] 1788820353.230468 punch from 0x349cf1 to 0x34b30f (0x161f bytes) 000260[1] 1788820353.266354 mapwrite 0x3df000 thru 0x3ece14 (0xde15 bytes) 000261[1] 1788820353.411215 trunc from 0x7c3958 to 0x9665cc (0x1a2c75 bytes) 000262[1] 1788820353.451480 read_OD 0x401000 thru 0x404fff (0x4000 bytes) 000264[0] 1788820353.484383 punch from 0x373d6f to 0x378eb9 (0x514b bytes) 000266[0] 1788820353.507433 mapwrite 0x638000 thru 0x63b870 (0x3871 bytes) 000268[1] 1788820353.639756 mapwrite 0x111000 thru 0x11bf8a (0xaf8b bytes) 000270[1] 1788820353.794065 trunc from 0x9665cd to 0x08d391 (0x8d923b bytes) 000272[1] 1788820353.855150 mapwrite 0x127000 thru 0x1362b0 (0xf2b1 bytes) 000276[1] 1788820354.023499 write_OD 0x8b1000 thru 0x8b7fff (0x7000 bytes) 000277[0] 1788820354.149841 mapread 0x414000 thru 0x41e9b7 (0xa9b8 bytes) 000278[1] 1788820354.195674 read_OD 0x893000 thru 0x89bfff (0x9000 bytes) 000279[1] 1788820354.209803 write_OD 0x8fb000 thru 0x900fff (0x6000 bytes) 000284[0] 1788820354.362048 write 0x8a1000 thru 0x8a29cb (0x19cc bytes) 000285[1] 1788820354.447700 punch from 0x2b1ee0 to 0x2b57af (0x38d0 bytes) 000287[0] 1788820354.474841 read 0x8c0000 thru 0x8ca08f (0xa090 bytes) 000288[1] 1788820354.501328 punch from 0x1b5a5b to 0x1bd655 (0x7bfb bytes) 000290[0] 1788820354.549215 mapwrite 0x098000 thru 0x0a3d26 (0xbd27 bytes) 000291[1] 1788820354.684905 mapwrite 0x281000 thru 0x28f15c (0xe15d bytes) 000292[1] 1788820354.789443 mapread 0x445000 thru 0x451331 (0xc332 bytes) 000294[0] 1788820354.858510 punch from 0x7c9ec1 to 0x7d5226 (0xb366 bytes) 000298[1] 1788820354.928680 mapread 0x74b000 thru 0x75780c (0xc80d bytes) 000303[0] 1788820355.004975 write 0x1ad000 thru 0x1b8d86 (0xbd87 bytes) 000304[1] 1788820355.047648 mapread 0x39c000 thru 0x3aafd2 (0xefd3 bytes) 000305[1] 1788820355.101096 punch from 0x56af35 to 0x573f23 (0x8fef bytes) 000306[0] 1788820355.116128 trunc from 0x901000 to 0x447eca (0x4b9135 bytes) 000307[0] 1788820355.153898 trunc from 0x447ecb to 0x9c10f6 (0x57922c bytes) 000309[1] 1788820355.195944 punch from 0x1e6e9a to 0x1f221f (0xb386 bytes) 000310[0] 1788820355.224633 write 0x0ba000 thru 0x0bd3de (0x33df bytes) 000312[1] 1788820355.256637 read_OD 0x74f000 thru 0x757fff (0x9000 bytes) 000314[0] 1788820355.286310 mapread 0x1ce000 thru 0x1ced7f (0xd80 bytes) 000315[1] 1788820355.294616 mapwrite 0x838000 thru 0x839241 (0x1242 bytes) 000316[1] 1788820355.373481 punch from 0x96c2b3 to 0x97840b (0xc159 bytes) 000317[1] 1788820355.380020 punch from 0x2af1dd to 0x2b8e62 (0x9c86 bytes) 000318[0] 1788820355.392424 punch from 0x8383b5 to 0x8480a8 (0xfcf4 bytes) 000319[0] 1788820355.418152 mapwrite 0x6a6000 thru 0x6b03fc (0xa3fd bytes) 000323[0] 1788820355.524625 read 0x63b000 thru 0x63ca35 (0x1a36 bytes) 000324[1] 1788820355.530579 trunc from 0x9c10f7 to 0x7d8841 (0x1e88b5 bytes) 000325[0] 1788820355.567256 mapread 0x3b6000 thru 0x3c236b (0xc36c bytes) 000327[1] 1788820355.627606 write_OD 0x5ba000 thru 0x5c4fff (0xb000 bytes) 000329[1] 1788820355.719581 read_OD 0x5eb000 thru 0x5f2fff (0x8000 bytes) 000332[1] 1788820355.735943 punch from 0x39b2ab to 0x3a1b9e (0x68f4 bytes) 000333[1] 1788820355.755457 write_OD 0x715000 thru 0x720fff (0xc000 bytes) 000335[0] 1788820355.873524 write 0x421000 thru 0x42dc3f (0xcc40 bytes) 000337[0] 1788820355.950022 mapwrite 0x917000 thru 0x91ada6 (0x3da7 bytes) 000340[0] 1788820356.108689 write 0x18a000 thru 0x18a382 (0x383 bytes) 000342[0] 1788820356.140957 mapread 0x37f000 thru 0x38758b (0x858c bytes) 000343[1] 1788820356.186184 punch from 0x21cab7 to 0x21fb2f (0x3079 bytes) 000345[1] 1788820356.234645 read_OD 0x0c5000 thru 0x0c5fff (0x1000 bytes) 000348[1] 1788820356.261866 mapwrite 0x637000 thru 0x639a5c (0x2a5d bytes) 000349[1] 1788820356.347409 punch from 0x2b6bfc to 0x2b7273 (0x678 bytes) 000351[1] 1788820356.384943 write_OD 0x878000 thru 0x879fff (0x2000 bytes) 000352[1] 1788820356.433677 mapread 0x63e000 thru 0x644417 (0x6418 bytes) 000354[1] 1788820356.447606 trunc from 0x91ada7 to 0x435acc (0x4e52da bytes) 000355[0] 1788820356.485459 trunc from 0x435acd to 0x90c187 (0x4d66bb bytes) 000356[1] 1788820356.520582 read_OD 0x7bc000 thru 0x7c6fff (0xb000 bytes) 000357[1] 1788820356.557258 trunc from 0x90c188 to 0x352ad3 (0x5b96b4 bytes) 000361[1] 1788820356.717534 read_OD 0x24d000 thru 0x256fff (0xa000 bytes) 000362[0] 1788820356.764772 mapread 0x1f1000 thru 0x1f8e4c (0x7e4d bytes) 000363[0] 1788820356.791782 trunc from 0x352ad4 to 0x0e25e5 (0x2704ee bytes) 000364[1] 1788820356.854295 mapwrite 0x936000 thru 0x936761 (0x762 bytes) 000367[1] 1788820357.086469 write_OD 0x49a000 thru 0x4a4fff (0xb000 bytes) 000368[1] 1788820357.226853 trunc from 0x936762 to 0x1366d3 (0x80008e bytes) 000369[0] 1788820357.275256 write 0x14a000 thru 0x1539da (0x99db bytes) 000372[1] 1788820357.328452 trunc from 0x1539db to 0x633289 (0x4df8af bytes) 000373[0] 1788820357.385597 read 0x0ef000 thru 0x0fb341 (0xc342 bytes) 000375[0] 1788820357.548255 punch from 0x03f7cf to 0x045be8 (0x641a bytes) 000377[0] 1788820357.619129 read 0x46f000 thru 0x47c8bd (0xd8be bytes) 000378[0] 1788820357.647243 mapread 0x0fa000 thru 0x0fa239 (0x23a bytes) 000379[1] 1788820357.654242 mapwrite 0x98b000 thru 0x99a7a6 (0xf7a7 bytes) 000380[0] 1788820357.884932 read 0x6db000 thru 0x6e4dc1 (0x9dc2 bytes) 000382[1] 1788820357.943370 write_OD 0x709000 thru 0x70dfff (0x5000 bytes) 000386[0] 1788820358.017283 punch from 0x1eea84 to 0x1ef5e9 (0xb66 bytes) 000387[0] 1788820358.033600 trunc from 0x99a7a7 to 0x5491d8 (0x4515ce bytes) 000388[0] 1788820358.103537 write 0x75c000 thru 0x764c6c (0x8c6d bytes) 000389[0] 1788820358.118369 mapread 0x6e2000 thru 0x6e73e0 (0x53e1 bytes) 000390[0] 1788820358.143689 punch from 0x070f4f to 0x07b0ea (0xa19c bytes) 000391[0] 1788820358.151842 mapwrite 0x07c000 thru 0x086b53 (0xab54 bytes) 000392[0] 1788820358.349340 write 0x45c000 thru 0x460caf (0x4cb0 bytes) 000394[1] 1788820358.359157 mapread 0x0b8000 thru 0x0b87ff (0x800 bytes) 000395[1] 1788820358.418060 trunc from 0x764c6d to 0x9a3242 (0x23e5d6 bytes) 000397[0] 1788820358.488154 punch from 0x46cef7 to 0x47c57f (0xf689 bytes) 000398[1] 1788820358.519555 mapread 0x3d6000 thru 0x3e230c (0xc30d bytes) 000399[1] 1788820358.604680 mapread 0x945000 thru 0x95092d (0xb92e bytes) 000402[0] 1788820358.663234 punch from 0x7dbe7f to 0x7eae81 (0xf003 bytes) 000404[0] 1788820358.684985 read 0x697000 thru 0x69f733 (0x8734 bytes) 000407[0] 1788820358.740631 mapread 0x4c0000 thru 0x4cc168 (0xc169 bytes) 000408[1] 1788820358.767014 punch from 0x7dd7e8 to 0x7ece44 (0xf65d bytes) 000409[1] 1788820358.802660 read_OD 0x8d4000 thru 0x8ddfff (0xa000 bytes) 000410[1] 1788820358.836667 write_OD 0x2be000 thru 0x2c5fff (0x8000 bytes) 000413[0] 1788820358.928973 trunc from 0x9a3243 to 0x102500 (0x8a0d42 bytes) 000414[0] 1788820359.009183 punch from 0x0ef0ef to 0x0f1f1a (0x2e2c bytes) 000417[0] 1788820359.066270 trunc from 0x102501 to 0x9c7d88 (0x8c5888 bytes) 000418[0] 1788820359.095276 write 0x6ef000 thru 0x6fd287 (0xe288 bytes) 000421[0] 1788820359.110914 punch from 0x0d878e to 0x0e66f1 (0xdf64 bytes) 000422[1] 1788820359.147941 punch from 0x406616 to 0x40a1ff (0x3bea bytes) 000423[1] 1788820359.229743 write_OD 0x473000 thru 0x47cfff (0xa000 bytes) 000425[1] 1788820359.334435 read_OD 0x12e000 thru 0x135fff (0x8000 bytes) 000426[1] 1788820359.344699 punch from 0x05caf4 to 0x05f655 (0x2b62 bytes) 000428[1] 1788820359.363223 punch from 0x8eb408 to 0x8eea1a (0x3613 bytes) 000429[1] 1788820359.382777 mapread 0x825000 thru 0x83163b (0xc63c bytes) 000430[1] 1788820359.447244 read_OD 0x274000 thru 0x27ffff (0xc000 bytes) 000431[1] 1788820359.472621 read_OD 0x7c5000 thru 0x7d0fff (0xc000 bytes) 000432[0] 1788820359.504445 write 0x79c000 thru 0x7a8471 (0xc472 bytes) 000434[0] 1788820359.603152 mapwrite 0x2a1000 thru 0x2a464f (0x3650 bytes) 000437[0] 1788820359.732651 punch from 0x44b60f to 0x44e03c (0x2a2e bytes) 000438[1] 1788820359.738686 mapread 0x18e000 thru 0x1981ca (0xa1cb bytes) 000440[0] 1788820359.777580 mapwrite 0x9f3000 thru 0x9ff3d9 (0xc3da bytes) 000442[1] 1788820359.976518 read_OD 0x6a6000 thru 0x6adfff (0x8000 bytes) 000443[0] 1788820360.019775 trunc from 0x9ff3da to 0x31c631 (0x6e2da8 bytes) 000444[0] 1788820360.069112 punch from 0x074e1c to 0x07c10a (0x72ef bytes) 000446[1] 1788820360.078334 read_OD 0x05e000 thru 0x066fff (0x9000 bytes) 000448[1] 1788820360.106501 mapread 0x0e3000 thru 0x0e388e (0x88f bytes) 000451[0] 1788820360.192956 mapwrite 0x5eb000 thru 0x5f8ba3 (0xdba4 bytes) 000454[0] 1788820360.373065 write 0x0e6000 thru 0x0ef3f0 (0x93f1 bytes) 000455[0] 1788820360.416489 read 0x543000 thru 0x54bdc6 (0x8dc7 bytes) 000456[1] 1788820360.440566 read_OD 0x42c000 thru 0x42ffff (0x4000 bytes) 000459[1] 1788820360.475224 punch from 0x05e283 to 0x061895 (0x3613 bytes) 000460[1] 1788820360.505469 punch from 0x5adce3 to 0x5af133 (0x1451 bytes) 000463[1] 1788820360.525327 mapread 0x3ae000 thru 0x3b9d6f (0xbd70 bytes) 000464[1] 1788820360.599258 read_OD 0x110000 thru 0x114fff (0x5000 bytes) 000465[1] 1788820360.621516 read_OD 0x1f1000 thru 0x1f8fff (0x8000 bytes) 000468[1] 1788820360.656762 mapread 0x50a000 thru 0x517c2c (0xdc2d bytes) 000469[1] 1788820360.722593 write_OD 0x3ec000 thru 0x3effff (0x4000 bytes) 000470[0] 1788820360.805156 read 0x3c4000 thru 0x3d1f1e (0xdf1f bytes) 000471[1] 1788820360.840774 read_OD 0x318000 thru 0x324fff (0xd000 bytes) 000472[0] 1788820360.873264 punch from 0x5b96d0 to 0x5bccac (0x35dd bytes) 000473[1] 1788820360.896052 write_OD 0x0df000 thru 0x0e4fff (0x6000 bytes) 000474[0] 1788820361.035078 mapread 0x07d000 thru 0x08acd0 (0xdcd1 bytes) 000475[1] 1788820361.085179 trunc from 0x5f8ba4 to 0x07b8f0 (0x57d2b3 bytes) 000476[0] 1788820361.171622 punch from 0x046867 to 0x05037c (0x9b16 bytes) 000478[0] 1788820361.191212 punch from 0x01fca9 to 0x02daa8 (0xde00 bytes) 000479[1] 1788820361.211985 punch from 0x00493a to 0x0059c8 (0x108f bytes) 000480[0] 1788820361.258464 write 0x187000 thru 0x18c2d9 (0x52da bytes) 000486[1] 1788820361.384945 mapwrite 0x054000 thru 0x06074c (0xc74d bytes) 000487[0] 1788820361.630988 trunc from 0x18c2da to 0x91edfc (0x792b23 bytes) 000491[1] 1788820361.698139 trunc from 0x91edfd to 0x6738b5 (0x2ab547 bytes) 000492[0] 1788820361.732225 read 0x5a2000 thru 0x5b1edf (0xfee0 bytes) 000493[0] 1788820361.779278 write 0x525000 thru 0x533f1c (0xef1d bytes) 000494[1] 1788820361.822588 write_OD 0x6a9000 thru 0x6aafff (0x2000 bytes) 000496[1] 1788820361.943666 mapread 0x5b9000 thru 0x5c245b (0x945c bytes) 000497[1] 1788820361.989626 punch from 0x5ac229 to 0x5b33c1 (0x7199 bytes) 000498[0] 1788820362.013398 punch from 0x04018d to 0x04afad (0xae21 bytes) 000499[0] 1788820362.070665 punch from 0x68aff6 to 0x692736 (0x7741 bytes) All operations completed A-OK! PASS 16a (52s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 18:32:46 (1788820366) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.324994 s, 25.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.509143 s, 16.5 MB/s Chance of close/open is 1 in 50 Seed set to 370 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole skipping zero size read 000005[1] 1788820371.157446 write 0x20decc thru 0x20f828 (0x195d bytes) 000008[1] 1788820371.217594 read 0x0e5910 thru 0x0e594b (0x3c bytes) 000009[1] 1788820371.241676 punch from 0x0dc303 to 0x0dd36e (0x106c bytes) 000010[0] 1788820371.260254 write 0x90d663 thru 0x90da00 (0x39e bytes) 000012[0] 1788820371.322883 mapread 0x0b1677 thru 0x0b33d5 (0x1d5f bytes) truncating to largest ever: 0x7ba9a9 000014[1] 1788820371.382496 trunc from 0x90da01 to 0x7ba9a8 (0x153058 bytes) 000015[0] 1788820371.452600 read 0x20503f thru 0x206179 (0x113b bytes) 000016[1] 1788820371.509165 punch from 0x6b7637 to 0x6b8e65 (0x182f bytes) 000017[0] 1788820371.527564 mapwrite 0x853077 thru 0x854071 (0xffb bytes) 000020[1] 1788820371.735523 punch from 0x4da73b to 0x4dadc3 (0x689 bytes) truncating to largest ever: 0x9357a0 000022[1] 1788820371.783472 trunc from 0x854072 to 0x93579f (0xe172e bytes) 000023[1] 1788820371.848674 punch from 0x578795 to 0x579bbd (0x1429 bytes) 000024[0] 1788820371.878699 mapwrite 0x46486e thru 0x465359 (0xaec bytes) 000026[1] 1788820372.151629 write 0x17398c thru 0x175163 (0x17d8 bytes) 000029[0] 1788820372.243120 write 0x1d7a3e thru 0x1d9913 (0x1ed6 bytes) 000030[0] 1788820372.299291 mapread 0x57dc12 thru 0x57e3de (0x7cd bytes) 000032[0] 1788820372.361072 punch from 0x6bbc35 to 0x6bd4c4 (0x1890 bytes) 000034[0] 1788820372.394511 punch from 0x489554 to 0x48a425 (0xed2 bytes) 000036[1] 1788820372.421355 write 0x7cdabb thru 0x7ce8cd (0xe13 bytes) 000038[1] 1788820372.474572 punch from 0x208d64 to 0x209cf9 (0xf96 bytes) 000041[0] 1788820372.561672 read 0x25c042 thru 0x25d411 (0x13d0 bytes) 000042[0] 1788820372.594179 punch from 0x9302b9 to 0x9306b1 (0x3f9 bytes) 000043[0] 1788820372.641030 read 0x30ee1b thru 0x3105d6 (0x17bc bytes) skipping zero size write 000045[1] 1788820372.678659 punch from 0x8208a8 to 0x820cb2 (0x40b bytes) 000048[0] 1788820372.699561 mapwrite 0x8b39cd thru 0x8b3b90 (0x1c4 bytes) 000052[0] 1788820372.854301 punch from 0x7acf7a to 0x7ad0bd (0x144 bytes) 000053[1] 1788820372.902325 read 0x3f3147 thru 0x3f3d56 (0xc10 bytes) 000058[1] 1788820372.996622 punch from 0x5b8531 to 0x5b8fad (0xa7d bytes) 000059[1] 1788820373.048494 read 0x09b373 thru 0x09b388 (0x16 bytes) 000060[1] 1788820373.053984 punch from 0x0e0dc9 to 0x0e2346 (0x157e bytes) 000061[1] 1788820373.071405 read 0x50d6b6 thru 0x50e252 (0xb9d bytes) 000063[0] 1788820373.085332 mapwrite 0x66fbe0 thru 0x6718b9 (0x1cda bytes) 000064[0] 1788820373.278693 trunc from 0x9357a0 to 0x8af6d4 (0x860cb bytes) 000067[0] 1788820373.386150 read 0x0fc0d3 thru 0x0fcc13 (0xb41 bytes) 000068[0] 1788820373.439351 write 0x722e28 thru 0x72372a (0x903 bytes) 000070[1] 1788820373.511187 write 0x6c811b thru 0x6c9767 (0x164d bytes) 000071[0] 1788820373.594893 mapread 0x6baecf thru 0x6bc7d8 (0x190a bytes) 000072[0] 1788820373.715180 mapwrite 0x5defcb thru 0x5dfa1b (0xa51 bytes) 000073[0] 1788820373.894008 trunc from 0x8af6d5 to 0x5f4f98 (0x2ba73c bytes) 000074[0] 1788820373.922037 mapread 0x1eb8da thru 0x1ed761 (0x1e88 bytes) 000075[0] 1788820374.008347 mapread 0x2f53f0 thru 0x2f73e1 (0x1ff2 bytes) 000077[1] 1788820374.048609 punch from 0x204bd6 to 0x2061a3 (0x15ce bytes) 000081[1] 1788820374.123556 mapread 0x23abf6 thru 0x23ad30 (0x13b bytes) 000082[0] 1788820374.148506 mapwrite 0x9f9fbe thru 0x9fa874 (0x8b7 bytes) 000083[1] 1788820374.253090 read 0x57cf4b thru 0x57e57b (0x1631 bytes) 000084[1] 1788820374.276439 punch from 0x672052 to 0x6727cc (0x77b bytes) 000085[0] 1788820374.303742 write 0x8e697c thru 0x8e707c (0x701 bytes) truncating to largest ever: 0x9ffe6a 000086[1] 1788820374.325097 trunc from 0x9fa875 to 0x9ffe69 (0x55f5 bytes) 000088[1] 1788820374.364657 trunc from 0x9ffe6a to 0x51fd46 (0x4e0123 bytes) 000090[0] 1788820374.432642 punch from 0x183032 to 0x183aac (0xa7b bytes) 000092[1] 1788820374.519459 trunc from 0x51fd47 to 0x13cf22 (0x3e2e24 bytes) 000095[1] 1788820374.635079 read 0x0635f3 thru 0x064bb9 (0x15c7 bytes) 000096[0] 1788820374.670991 punch from 0x062de4 to 0x063e14 (0x1031 bytes) 000097[1] 1788820374.704218 write 0x947fbd thru 0x948249 (0x28d bytes) 000098[1] 1788820374.717600 punch from 0x0b44e4 to 0x0b4f9d (0xaba bytes) 000099[1] 1788820374.726362 mapwrite 0x4e0d58 thru 0x4e2084 (0x132d bytes) 000100[0] 1788820374.820641 mapread 0x6e54fb thru 0x6e65c7 (0x10cd bytes) 000101[1] 1788820374.881970 mapwrite 0x786205 thru 0x787d59 (0x1b55 bytes) 000102[0] 1788820374.990453 mapread 0x4071db thru 0x409194 (0x1fba bytes) 000103[1] 1788820375.057150 mapwrite 0x9c5cb3 thru 0x9c6ff7 (0x1345 bytes) 000105[0] 1788820375.167161 mapread 0x8201e7 thru 0x821b95 (0x19af bytes) 000106[0] 1788820375.206552 mapread 0x22c920 thru 0x22e07c (0x175d bytes) 000107[1] 1788820375.245343 mapread 0x8d7ba2 thru 0x8d8340 (0x79f bytes) 000109[0] 1788820375.273023 write 0x8464db thru 0x8480d6 (0x1bfc bytes) 000112[1] 1788820375.351201 punch from 0x20b59d to 0x20c382 (0xde6 bytes) 000113[0] 1788820375.386515 punch from 0x638cce to 0x639a0a (0xd3d bytes) 000114[1] 1788820375.421497 mapwrite 0x098b05 thru 0x099e3b (0x1337 bytes) 000118[1] 1788820375.597157 read 0x44a434 thru 0x44ad80 (0x94d bytes) 000119[1] 1788820375.611575 punch from 0x0ef63f to 0x0f1193 (0x1b55 bytes) 000120[1] 1788820375.617689 mapread 0x0b602c thru 0x0b6b11 (0xae6 bytes) 000122[1] 1788820375.649322 write 0x690669 thru 0x6923ae (0x1d46 bytes) 000123[0] 1788820375.692443 read 0x3cce28 thru 0x3ce55b (0x1734 bytes) 000126[1] 1788820375.729595 trunc from 0x9c6ff8 to 0x111c5e (0x8b5399 bytes) 000128[0] 1788820375.765306 punch from 0x049bd4 to 0x04aa77 (0xea4 bytes) 000129[0] 1788820375.797081 mapread 0x00bc42 thru 0x00c47f (0x83e bytes) 000130[0] 1788820375.827093 read 0x0ae9c1 thru 0x0b01ed (0x182d bytes) 000132[1] 1788820375.900239 punch from 0x093ae9 to 0x094b04 (0x101c bytes) 000134[0] 1788820375.945977 mapwrite 0x26179d thru 0x262986 (0x11ea bytes) 000136[0] 1788820376.125030 write 0x8fb96c thru 0x8fc289 (0x91e bytes) 000137[0] 1788820376.141238 punch from 0x090bf9 to 0x091cda (0x10e2 bytes) 000138[0] 1788820376.151660 trunc from 0x8fc28a to 0x0bf6da (0x83cbaf bytes) 000141[0] 1788820376.250831 trunc from 0x0bf6db to 0x5fc007 (0x53c92d bytes) 000142[1] 1788820376.278757 write 0x23e907 thru 0x23f253 (0x94d bytes) 000143[1] 1788820376.325054 read 0x49f07d thru 0x49f150 (0xd4 bytes) 000144[0] 1788820376.365425 write 0x1876ec thru 0x188004 (0x919 bytes) 000145[0] 1788820376.497508 mapread 0x0d4dbd thru 0x0d68e9 (0x1b2d bytes) 000146[0] 1788820376.561835 write 0x4b5803 thru 0x4b64d9 (0xcd7 bytes) 000147[0] 1788820376.690844 read 0x32b0ab thru 0x32bd48 (0xc9e bytes) 000148[0] 1788820376.734189 read 0x02824c thru 0x028c32 (0x9e7 bytes) 000149[0] 1788820376.744499 mapwrite 0x648879 thru 0x6490ba (0x842 bytes) 000151[0] 1788820376.929273 mapread 0x3cdb29 thru 0x3cee7f (0x1357 bytes) 000153[0] 1788820376.975624 mapwrite 0x193c7c thru 0x195ac5 (0x1e4a bytes) 000154[0] 1788820377.166265 trunc from 0x6490bb to 0x29d11f (0x3abf9b bytes) 000155[0] 1788820377.267426 mapwrite 0x10f52b thru 0x10f83a (0x310 bytes) 000159[1] 1788820377.355806 trunc from 0x29d120 to 0x4deedd (0x241dbe bytes) 000161[1] 1788820377.427354 punch from 0x0517a2 to 0x051c77 (0x4d6 bytes) 000163[0] 1788820377.494434 punch from 0x087226 to 0x0887c1 (0x159c bytes) 000166[1] 1788820377.642192 read 0x309b9e thru 0x30aa5d (0xec0 bytes) 000168[1] 1788820377.700758 write 0x1d85bf thru 0x1d9a6b (0x14ad bytes) 000169[1] 1788820377.734391 mapread 0x297629 thru 0x297e3f (0x817 bytes) 000171[1] 1788820377.769374 mapwrite 0x598317 thru 0x598e07 (0xaf1 bytes) 000172[1] 1788820377.891338 mapread 0x3b5891 thru 0x3b6ca2 (0x1412 bytes) 000173[1] 1788820377.930467 read 0x360708 thru 0x362128 (0x1a21 bytes) 000174[1] 1788820377.978819 read 0x43499a thru 0x43578d (0xdf4 bytes) 000177[0] 1788820378.050094 trunc from 0x598e08 to 0x3e331d (0x1b5aea bytes) 000178[0] 1788820378.141622 trunc from 0x3e331e to 0x92adac (0x547a8f bytes) 000179[1] 1788820378.186418 mapwrite 0x07c2bf thru 0x07dc12 (0x1954 bytes) 000180[0] 1788820378.273563 trunc from 0x92adad to 0x09b8be (0x88f4ee bytes) 000181[0] 1788820378.309510 mapwrite 0x6a0849 thru 0x6a093b (0xf3 bytes) 000182[1] 1788820378.441008 punch from 0x4a1209 to 0x4a3199 (0x1f91 bytes) 000183[1] 1788820378.484120 punch from 0x3c66c5 to 0x3c676d (0xa9 bytes) 000184[1] 1788820378.525417 mapwrite 0x6b6e18 thru 0x6b7fd6 (0x11bf bytes) 000188[0] 1788820378.712216 mapread 0x0ddc85 thru 0x0deb42 (0xebe bytes) 000189[0] 1788820378.777351 write 0x7c2a07 thru 0x7c3fc2 (0x15bc bytes) 000192[1] 1788820378.792181 mapwrite 0x2406c2 thru 0x241706 (0x1045 bytes) 000195[1] 1788820379.001283 read 0x56a2e3 thru 0x56bbc6 (0x18e4 bytes) 000196[1] 1788820379.044694 mapread 0x2c1080 thru 0x2c29b3 (0x1934 bytes) 000197[0] 1788820379.067635 punch from 0x4aa04e to 0x4aac0a (0xbbd bytes) 000198[1] 1788820379.098362 trunc from 0x7c3fc3 to 0x60a8e0 (0x1b96e2 bytes) 000201[1] 1788820379.216846 write 0x38628b thru 0x387dad (0x1b23 bytes) 000202[1] 1788820379.263737 punch from 0x56ae0e to 0x56c172 (0x1365 bytes) 000206[1] 1788820379.383121 punch from 0x45148d to 0x4524bf (0x1033 bytes) 000208[1] 1788820379.438094 read 0x4d559f thru 0x4d5e3b (0x89d bytes) 000209[1] 1788820379.491051 mapwrite 0x366624 thru 0x36752a (0xf07 bytes) 000210[0] 1788820379.653672 mapwrite 0x164d4d thru 0x16551f (0x7d3 bytes) 000211[1] 1788820379.792597 punch from 0x467623 to 0x4676cb (0xa9 bytes) 000213[1] 1788820379.840487 trunc from 0x60a8e1 to 0x64f90e (0x4502e bytes) 000215[0] 1788820379.872645 mapwrite 0x10428c thru 0x105342 (0x10b7 bytes) 000216[1] 1788820379.989698 punch from 0x0f6684 to 0x0f6abc (0x439 bytes) 000217[0] 1788820380.008100 punch from 0x3f989c to 0x3f98c6 (0x2b bytes) 000218[1] 1788820380.036226 punch from 0x1fbb16 to 0x1fd6a5 (0x1b90 bytes) 000219[0] 1788820380.057066 mapwrite 0x32a726 thru 0x32be01 (0x16dc bytes) 000222[1] 1788820380.269517 read 0x57562e thru 0x576c92 (0x1665 bytes) 000223[0] 1788820380.289981 mapread 0x24b904 thru 0x24c995 (0x1092 bytes) 000224[1] 1788820380.332480 trunc from 0x64f90f to 0x777fe7 (0x1286d9 bytes) 000228[1] 1788820380.448699 punch from 0x36a4cb to 0x36a6d8 (0x20e bytes) 000230[0] 1788820380.457322 mapwrite 0x0594b2 thru 0x05aa91 (0x15e0 bytes) 000232[1] 1788820380.569082 trunc from 0x777fe8 to 0x358163 (0x41fe84 bytes) 000234[1] 1788820380.638356 trunc from 0x358164 to 0x14259b (0x215bc8 bytes) 000236[1] 1788820380.680249 mapread 0x0ea411 thru 0x0eafa1 (0xb91 bytes) 000237[0] 1788820380.686546 write 0x965c07 thru 0x966cda (0x10d4 bytes) 000239[1] 1788820380.729847 trunc from 0x966cdb to 0x38c997 (0x5da343 bytes) 000240[1] 1788820380.795517 punch from 0x2d6c15 to 0x2d6f15 (0x301 bytes) 000245[1] 1788820380.844317 write 0x605adf thru 0x6071af (0x16d1 bytes) 000246[1] 1788820380.867058 mapread 0x1c51b9 thru 0x1c5798 (0x5e0 bytes) 000248[1] 1788820380.910102 write 0x8dedcb thru 0x8e0a4b (0x1c81 bytes) 000249[1] 1788820380.914429 write 0x8ec958 thru 0x8ee1fb (0x18a4 bytes) 000251[0] 1788820380.940087 read 0x3b514b thru 0x3b56d0 (0x586 bytes) 000253[1] 1788820380.996082 trunc from 0x8ee1fc to 0x919a32 (0x2b837 bytes) 000256[0] 1788820381.098049 trunc from 0x919a33 to 0x7a96f5 (0x17033d bytes) 000257[1] 1788820381.190518 mapread 0x4bf548 thru 0x4c0575 (0x102e bytes) 000258[1] 1788820381.231883 write 0x4232d8 thru 0x424788 (0x14b1 bytes) 000259[0] 1788820381.280451 write 0x6e6813 thru 0x6e870f (0x1efd bytes) 000261[1] 1788820381.375881 mapread 0x520a57 thru 0x521c17 (0x11c1 bytes) 000262[0] 1788820381.435212 mapwrite 0x19f7a5 thru 0x1a016c (0x9c8 bytes) 000263[1] 1788820381.616122 punch from 0x25a931 to 0x25af36 (0x606 bytes) 000266[0] 1788820381.704688 write 0x1644b9 thru 0x1658e9 (0x1431 bytes) 000267[1] 1788820381.753677 trunc from 0x7a96f6 to 0x74bfb8 (0x5d73d bytes) 000268[0] 1788820381.836165 write 0x18f5ad thru 0x190d34 (0x1788 bytes) 000270[0] 1788820381.893400 read 0x5207e1 thru 0x521a26 (0x1246 bytes) 000272[1] 1788820381.952916 punch from 0x740b6e to 0x74189c (0xd2f bytes) 000273[0] 1788820381.999869 punch from 0x0e4636 to 0x0e5645 (0x1010 bytes) 000274[1] 1788820382.026063 mapwrite 0x77530f thru 0x7759cc (0x6be bytes) 000276[1] 1788820382.264515 read 0x27554f thru 0x2769af (0x1461 bytes) 000278[1] 1788820382.304512 mapread 0x586388 thru 0x5877ec (0x1465 bytes) 000281[1] 1788820382.318365 write 0x61e37c thru 0x62036d (0x1ff2 bytes) 000282[1] 1788820382.327159 punch from 0x6ab007 to 0x6ab161 (0x15b bytes) 000283[0] 1788820382.335787 trunc from 0x7759cd to 0x3a4de5 (0x3d0be7 bytes) 000284[1] 1788820382.457219 mapwrite 0x5366e6 thru 0x537a63 (0x137e bytes) 000287[1] 1788820382.733234 trunc from 0x537a64 to 0x20b2d4 (0x32c78f bytes) 000288[0] 1788820382.775857 punch from 0x1f3173 to 0x1f4e06 (0x1c94 bytes) 000289[1] 1788820382.833838 mapread 0x13adcb thru 0x13b4ed (0x723 bytes) 000291[1] 1788820382.863046 trunc from 0x20b2d5 to 0x09d929 (0x16d9ab bytes) 000292[1] 1788820382.919440 punch from 0x0900f2 to 0x090888 (0x797 bytes) 000293[1] 1788820382.929404 trunc from 0x09d92a to 0x0927a8 (0xb181 bytes) 000295[0] 1788820382.987863 mapread 0x01c243 thru 0x01cc8f (0xa4d bytes) 000299[0] 1788820383.066288 read 0x0579fe thru 0x059961 (0x1f64 bytes) 000300[0] 1788820383.091319 write 0x4f768f thru 0x4f7717 (0x89 bytes) 000305[0] 1788820383.210124 punch from 0x4a97b4 to 0x4a9f4c (0x799 bytes) 000306[0] 1788820383.221559 trunc from 0x4f7718 to 0x21a65c (0x2dd0bb bytes) 000308[1] 1788820383.304823 write 0x5bd253 thru 0x5be386 (0x1134 bytes) 000309[0] 1788820383.340424 mapread 0x48de92 thru 0x48fc02 (0x1d71 bytes) 000311[0] 1788820383.422558 trunc from 0x5be387 to 0x0181ee (0x5a6198 bytes) 000314[1] 1788820383.549881 mapread 0x00572a thru 0x00732e (0x1c05 bytes) 000319[1] 1788820383.628498 mapread 0x0039ac thru 0x0043f9 (0xa4e bytes) 000321[0] 1788820383.670773 write 0x37209a thru 0x373821 (0x1788 bytes) 000322[0] 1788820383.699493 read 0x11d550 thru 0x11e5be (0x106f bytes) 000323[0] 1788820383.727534 read 0x293fd3 thru 0x295af3 (0x1b21 bytes) 000324[1] 1788820383.728153 write 0x87befc thru 0x87d353 (0x1458 bytes) 000325[1] 1788820383.764823 write 0x6bc93a thru 0x6bda13 (0x10da bytes) 000327[0] 1788820383.781189 punch from 0x54b52e to 0x54ba7d (0x550 bytes) 000328[0] 1788820383.827570 trunc from 0x87d354 to 0x71a8b7 (0x162a9c bytes) 000330[0] 1788820383.896589 trunc from 0x71a8b8 to 0x727b30 (0xd279 bytes) 000332[0] 1788820383.928247 punch from 0x52114f to 0x521aae (0x960 bytes) 000333[0] 1788820383.980944 mapread 0x226085 thru 0x22710b (0x1087 bytes) 000334[1] 1788820384.096520 mapwrite 0x86140a thru 0x86266f (0x1266 bytes) 000336[1] 1788820384.253273 trunc from 0x862670 to 0x1bce3c (0x6a5833 bytes) 000338[0] 1788820384.323468 mapwrite 0x943783 thru 0x944831 (0x10af bytes) 000339[0] 1788820384.546808 mapread 0x462aa0 thru 0x462ee7 (0x448 bytes) 000340[1] 1788820384.571892 trunc from 0x944832 to 0x492554 (0x4b22dd bytes) 000341[1] 1788820384.636047 write 0x7c079b thru 0x7c0bcd (0x433 bytes) 000342[0] 1788820384.639388 punch from 0x08e5d7 to 0x08f110 (0xb3a bytes) 000343[0] 1788820384.693114 mapread 0x59161a thru 0x592706 (0x10ed bytes) 000345[0] 1788820384.763990 trunc from 0x7c0bce to 0x4ffb75 (0x2c1058 bytes) 000347[0] 1788820384.816670 punch from 0x089c52 to 0x089da5 (0x154 bytes) 000348[0] 1788820384.822393 punch from 0x3095d8 to 0x30a6bb (0x10e4 bytes) 000349[1] 1788820384.865449 mapread 0x45df65 thru 0x45f48f (0x152b bytes) 000350[1] 1788820384.945605 mapread 0x1fa08e thru 0x1fb51e (0x1491 bytes) 000352[1] 1788820385.002096 read 0x45f2e0 thru 0x4601cf (0xef0 bytes) 000353[0] 1788820385.003012 trunc from 0x4ffb76 to 0x8c95ef (0x3c9a7a bytes) 000354[0] 1788820385.102456 trunc from 0x8c95f0 to 0x1d7708 (0x6f1ee7 bytes) 000355[0] 1788820385.170458 mapwrite 0x14ab1b thru 0x14c19e (0x1684 bytes) 000356[1] 1788820385.283015 mapread 0x1745e0 thru 0x176112 (0x1b33 bytes) 000357[1] 1788820385.325042 trunc from 0x1d7709 to 0x39fc9b (0x1c8593 bytes) 000359[1] 1788820385.438911 trunc from 0x39fc9c to 0x3f4641 (0x549a6 bytes) 000360[0] 1788820385.487419 punch from 0x03ce30 to 0x03cf55 (0x126 bytes) 000362[0] 1788820385.529575 mapwrite 0x7be673 thru 0x7bee4d (0x7db bytes) 000364[1] 1788820385.679869 read 0x108aa2 thru 0x10a271 (0x17d0 bytes) 000365[1] 1788820385.739677 trunc from 0x7bee4e to 0x220d1f (0x59e12e bytes) 000367[1] 1788820385.805410 trunc from 0x220d20 to 0x4f965b (0x2d893c bytes) 000368[0] 1788820385.890130 mapread 0x0bc428 thru 0x0bc998 (0x571 bytes) 000370[0] 1788820385.958584 read 0x479f96 thru 0x47bb83 (0x1bee bytes) 000371[0] 1788820385.985453 trunc from 0x4f965c to 0x725aa9 (0x22c44e bytes) 000372[1] 1788820386.052545 write 0x023ff8 thru 0x025075 (0x107e bytes) 000378[1] 1788820386.166195 mapwrite 0x361468 thru 0x362ab3 (0x164c bytes) 000380[0] 1788820386.253986 punch from 0x4a3c50 to 0x4a3e3c (0x1ed bytes) 000383[0] 1788820386.308077 mapwrite 0x0d9fb1 thru 0x0dbf1e (0x1f6e bytes) 000384[0] 1788820386.432051 punch from 0x199e2c to 0x19ac41 (0xe16 bytes) 000387[1] 1788820386.455149 write 0x2af77a thru 0x2b1511 (0x1d98 bytes) 000388[0] 1788820386.507236 read 0x043e70 thru 0x0450bd (0x124e bytes) 000389[1] 1788820386.538687 mapwrite 0x12ef56 thru 0x130441 (0x14ec bytes) 000391[1] 1788820386.683106 read 0x6e8927 thru 0x6e9fd3 (0x16ad bytes) 000392[1] 1788820386.722454 trunc from 0x725aaa to 0x4cee99 (0x256c10 bytes) 000395[0] 1788820386.896324 punch from 0x45115f to 0x4520ac (0xf4e bytes) 000397[0] 1788820387.007248 trunc from 0x4cee9a to 0x34e56d (0x18092c bytes) 000398[1] 1788820387.078454 write 0x08bb52 thru 0x08c5a4 (0xa53 bytes) 000399[1] 1788820387.129832 punch from 0x2143b7 to 0x2156b7 (0x1301 bytes) 000403[0] 1788820387.226550 write 0x45e7a8 thru 0x45e9d0 (0x229 bytes) 000404[1] 1788820387.247256 trunc from 0x45e9d1 to 0x5875b1 (0x128be1 bytes) 000405[1] 1788820387.309583 trunc from 0x5875b2 to 0x7d92c6 (0x251d15 bytes) 000406[1] 1788820387.349628 read 0x58a177 thru 0x58b369 (0x11f3 bytes) 000408[1] 1788820387.359980 mapwrite 0x6091c4 thru 0x6096d2 (0x50f bytes) 000412[1] 1788820387.445200 mapwrite 0x51a4fb thru 0x51a71a (0x220 bytes) 000413[1] 1788820387.487370 read 0x444c17 thru 0x4454d4 (0x8be bytes) 000414[0] 1788820387.493461 mapwrite 0x306651 thru 0x3082cf (0x1c7f bytes) 000417[0] 1788820387.632482 write 0x0ba229 thru 0x0bb6b7 (0x148f bytes) 000418[0] 1788820387.659070 read 0x20f728 thru 0x21014a (0xa23 bytes) 000419[1] 1788820387.682159 trunc from 0x7d92c7 to 0x58d015 (0x24c2b1 bytes) 000422[0] 1788820387.800262 trunc from 0x58d016 to 0x890b01 (0x303aec bytes) 000423[1] 1788820387.857092 punch from 0x58a92f to 0x58ab86 (0x258 bytes) 000424[1] 1788820387.896331 write 0x1c7ba5 thru 0x1c9834 (0x1c90 bytes) 000426[1] 1788820387.953301 mapwrite 0x62df4e thru 0x62eb78 (0xc2b bytes) 000427[0] 1788820388.032038 punch from 0x05217a to 0x0535af (0x1436 bytes) 000428[1] 1788820388.089929 mapread 0x7dfa80 thru 0x7e08cf (0xe50 bytes) 000429[1] 1788820388.139108 mapread 0x4e3c03 thru 0x4e59cf (0x1dcd bytes) 000430[0] 1788820388.215940 punch from 0x68d198 to 0x68d540 (0x3a9 bytes) 000431[1] 1788820388.233400 mapwrite 0x0897e7 thru 0x08a178 (0x992 bytes) 000433[0] 1788820388.373116 mapwrite 0x4f0523 thru 0x4f1d1f (0x17fd bytes) 000434[0] 1788820388.459200 trunc from 0x890b02 to 0x6432c7 (0x24d83a bytes) 000435[1] 1788820388.520689 mapwrite 0x244055 thru 0x2457ea (0x1796 bytes) 000436[0] 1788820388.689680 punch from 0x12c219 to 0x12de55 (0x1c3d bytes) 000438[0] 1788820388.761108 mapwrite 0x7829a5 thru 0x78308e (0x6ea bytes) 000439[0] 1788820388.927638 punch from 0x2bc28d to 0x2bdf62 (0x1cd6 bytes) 000440[1] 1788820388.934533 read 0x3ae33e thru 0x3b023f (0x1f02 bytes) 000442[0] 1788820389.065310 trunc from 0x78308f to 0x5687fc (0x21a892 bytes) 000443[0] 1788820389.107880 punch from 0x4cb567 to 0x4cc920 (0x13ba bytes) 000446[0] 1788820389.116424 trunc from 0x5687fd to 0x6e0082 (0x177886 bytes) 000447[1] 1788820389.140306 punch from 0x67b59c to 0x67cd8a (0x17ef bytes) 000449[0] 1788820389.209670 mapwrite 0x8f9117 thru 0x8fa24a (0x1134 bytes) 000453[1] 1788820389.445686 read 0x55b3ad thru 0x55c475 (0x10c9 bytes) 000454[0] 1788820389.466849 mapread 0x6c4813 thru 0x6c4b78 (0x366 bytes) 000455[1] 1788820389.484214 trunc from 0x8fa24b to 0x90c6c1 (0x12477 bytes) 000456[1] 1788820389.547406 trunc from 0x90c6c2 to 0x3d72d6 (0x5353eb bytes) 000457[0] 1788820389.582337 mapread 0x359bf5 thru 0x359c58 (0x64 bytes) 000458[0] 1788820389.625368 mapwrite 0x473dcb thru 0x473f0f (0x145 bytes) 000462[0] 1788820389.822214 punch from 0x0891ec to 0x08a1b7 (0xfcc bytes) 000463[1] 1788820389.865773 write 0x2600c4 thru 0x2617d7 (0x1714 bytes) 000464[0] 1788820389.938489 mapread 0x1fe09f thru 0x1ff862 (0x17c4 bytes) 000466[0] 1788820390.010418 write 0x265f87 thru 0x267aa2 (0x1b1c bytes) 000468[1] 1788820390.056263 mapread 0x1b9ef6 thru 0x1bb78f (0x189a bytes) 000469[0] 1788820390.121260 trunc from 0x473f10 to 0x872d91 (0x3fee82 bytes) 000471[1] 1788820390.188125 mapwrite 0x90de2f thru 0x90e2b7 (0x489 bytes) 000472[1] 1788820390.315270 punch from 0x6333b8 to 0x633908 (0x551 bytes) 000473[1] 1788820390.329261 read 0x266416 thru 0x266b38 (0x723 bytes) 000474[0] 1788820390.356641 punch from 0x383f67 to 0x38416a (0x204 bytes) 000475[1] 1788820390.390195 write 0x7412d5 thru 0x741fcf (0xcfb bytes) 000477[1] 1788820390.421212 mapread 0x7400a6 thru 0x740cf5 (0xc50 bytes) 000480[1] 1788820390.486829 read 0x152ccd thru 0x154c93 (0x1fc7 bytes) 000481[1] 1788820390.523970 read 0x7d5424 thru 0x7d72d8 (0x1eb5 bytes) 000482[1] 1788820390.553670 trunc from 0x90e2b8 to 0x717cec (0x1f65cb bytes) 000485[1] 1788820390.628674 write 0x193ccd thru 0x1947a3 (0xad7 bytes) 000486[0] 1788820390.640098 mapread 0x21088f thru 0x212284 (0x19f6 bytes) 000488[1] 1788820390.680916 read 0x5fa8b8 thru 0x5fbf11 (0x165a bytes) 000489[1] 1788820390.697053 write 0x2ec2e0 thru 0x2edafa (0x181b bytes) 000490[1] 1788820390.718009 read 0x4604e9 thru 0x4615f9 (0x1111 bytes) 000493[0] 1788820390.754462 punch from 0x6b4070 to 0x6b5e7d (0x1e0e bytes) 000494[0] 1788820390.794122 mapread 0x42341c thru 0x42394c (0x531 bytes) 000495[0] 1788820390.859917 write 0x404a78 thru 0x40543b (0x9c4 bytes) 000496[0] 1788820390.903164 mapwrite 0x04b918 thru 0x04d14c (0x1835 bytes) 000499[0] 1788820391.031126 write 0x07a71c thru 0x07bc2e (0x1513 bytes) 000500[0] 1788820391.039431 mapread 0x0d176e thru 0x0d30a6 (0x1939 bytes) All operations completed A-OK! PASS 16b (29s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 18:33:16 (1788820396) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 18:33:19 (1788820399) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.323096 s, 26.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.282007 s, 29.7 MB/s ldlm.namespaces.MGC192.168.203.103@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9131c3268800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9131e025f800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9131c3268800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9131e025f800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9131c3268800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9131e025f800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.7472 s, 8.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.49968 s, 23.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.8642 s, 36.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.05309 s, 17.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.34641 s, 11.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.45015 s, 30.4 MB/s PASS 16d (53s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 18:34:12 (1788820452) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.397819 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144859 s, 7.2 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 18:34:20 (1788820460) TIMEOUT PASS 16f (30s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 18:34:50 (1788820490) TIMEOUT PASS 16g (28s) == sanityn test 16h: mmap read after truncate file ======= 18:35:18 (1788820518) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (5s) == sanityn test 16i: read after truncate file ============ 18:35:23 (1788820523) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (7s) == sanityn test 16j: race dio with buffered i/o ========== 18:35:30 (1788820530) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.9765 s, 13.3 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.29755 s, 20.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.50507 s, 10.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.87081 s, 14.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.50726 s, 10.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.11013 s, 12.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.76461 s, 9.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.842832 s, 31.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.13596 s, 23.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.47329 s, 10.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.51764 s, 10.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.05005 s, 12.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.20074 s, 11.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.6869 s, 9.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.68716 s, 9.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.12764 s, 8.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.50461 s, 7.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.5254 s, 7.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.60161 s, 7.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (35s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 18:36:06 (1788820566) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Mon Sep 7 18:37:01 EDT 2026 All operations completed A-OK! + fsx 2 done: rc=0 Mon Sep 7 18:37:01 EDT 2026 All operations completed A-OK! + fsx 3 done: rc=0 Mon Sep 7 18:37:01 EDT 2026 + fsx 4 done: rc=0 Mon Sep 7 18:37:02 EDT 2026 + fsx 5 done: rc=0 Mon Sep 7 18:37:02 EDT 2026 PASS 16k (64s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 18:37:09 (1788820629) fail_loc=0x8000030a PASS 17 (10s) == sanityn test 18: mmap sanity check =========================================================================================== 18:37:19 (1788820639) mmap test1: basic mmap operation (PASS, 0.13828s) mmap test2: MAP_PRIVATE not write back (PASS, 0.078935s) mmap test3: concurrent mmap ops on two nodes (PASS, 11.317s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.5735s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.19493s) mmap test6: check mmap write/read content on two nodes (PASS, 0.39844s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.3584s) mmap test11: multi-thread shared mmap (PASS, 1.235s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1086s) PASS 18 (31s) == sanityn test 19: test concurrent uncached read races ========================================================================= 18:37:50 (1788820670) oleg303-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg303-client: oleg303-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 18:37:55 (1788820675) PASS 20 (7s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 18:38:03 (1788820683) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (6s) == sanityn test 23: others should see updated atime while another read============================================================== 18:38:09 (1788820689) now is 1788820691 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 new atime is 1788820752 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 18:39:19 (1788820759) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 18:39:27 (1788820767) PASS 24b (8s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 18:39:35 (1788820775) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (9s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 18:39:44 (1788820784) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (4s) == sanityn test 26a: allow mtime to get older ============ 18:39:48 (1788820788) utime: good mknod times 1788820789 <= 1788820790 <= 1788820790 for /mnt/lustre/f26a utime: good mknod times 1788820789 <= 1788820790 <= 1788820790 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (9s) == sanityn test 26b: sync mtime between ost and mds ====== 18:39:57 (1788820797) PASS 26b (9s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 18:40:06 (1788820806) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-07 18:40:08.000000000 -0400 Birth: 2026-09-07 18:40:09.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-07 18:40:08.000000000 -0400 Birth: 2026-09-07 18:40:09.000000000 -0400 PASS 26c (10s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 18:40:18 (1788820818) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 18:40:28 (1788820828) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 18:40:36 (1788820836) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg303-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.103@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (20s) == sanityn test 31r: open-rename(replace) race =========== 18:40:56 (1788820856) -rw-r--r-- 1 root root 0 Sep 7 18:40 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (9s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 18:41:06 (1788820866) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43165 Terminated tail -f $tfile PASS 31s (7s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 18:41:13 (1788820873) PASS 31t (6s) == sanityn test 32b: lockless i/o ======================== 18:41:20 (1788820880) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (2s) == sanityn test 32c: contention detection ================ 18:41:23 (1788820883) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.13577 s, 46.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.17181 s, 44.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.76253 s, 29.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.9563 s, 26.8 MB/s contention_seconds 4, contention_events 17 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.94899 s, 26.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.28805 s, 22.9 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.10213 s, 47.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.12321 s, 46.7 MB/s PASS 32c (25s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 18:41:50 (1788820910) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 18:41:53 (1788820913) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 18:41:56 (1788820916) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 18:41:59 (1788820919) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 18:42:02 (1788820922) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg303-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9131c3268800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9131c3268800.ost_server_uuid in IDLE state after 0 sec oleg303-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9131c3268800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9131c3268800.ost_server_uuid in IDLE state after 0 sec oleg303-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9131c3268800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9131c3268800.ost_server_uuid in IDLE state after 0 sec oleg303-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9131c3268800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9131c3268800.ost_server_uuid in IDLE state after 0 sec oleg303-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9131c3268800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9131c3268800.ost_server_uuid in IDLE state after 0 sec oleg303-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9131c3268800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9131c3268800.ost_server_uuid in IDLE state after 0 sec PASS 34 (148s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 18:44:30 (1788821070) fail_loc=0x80000317 Race attempt 0 Wait for 50468 50486 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1855: 50468 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.103@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.203.103@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 - open/close 600 (time 1788821089.76 total 10.01 last 59.96) - open/close 1241 (time 1788821099.76 total 20.01 last 64.05) - open/close 1901 (time 1788821109.76 total 30.01 last 65.99) - open/close 2416 (time 1788821119.77 total 40.02 last 51.47) - open/close 2936 (time 1788821129.79 total 50.04 last 51.88) - open/close 3457 (time 1788821139.80 total 60.05 last 52.07) - open/close 3843 (time 1788821149.81 total 70.06 last 38.57) total: 4000 open/close in 73.40 seconds: 54.49 ops/second PASS 35 (83s) == sanityn test 36: handle ESTALE/open-unlink correctly == 18:45:53 (1788821153) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg303-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.837952 s, 62.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7422976) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.18734 s, 44.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7439360) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.768856 s, 68.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7492608) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.810944 s, 64.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7493632) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.624234 s, 84.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7490560) after_dd(7432192) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.952571 s, 55.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7497728) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.549074 s, 95.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7490560) after_dd(7430144) after(7481344) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.561835 s, 93.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7481344) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.538985 s, 97.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7490560) after_dd(7438336) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.917049 s, 57.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7490560) after_dd(7427072) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.878835 s, 59.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(10) *** before(7496704) after_dd(7443456) after(7494656) PASS 36 (318s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 18:51:11 (1788821471) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 - create 1198 (time 1788821485.12 total 10.01 last 119.73) - create 2186 (time 1788821495.13 total 20.01 last 98.72) - create 3416 (time 1788821505.14 total 30.02 last 122.94) - create 4439 (time 1788821515.14 total 40.02 last 102.24) - create 5418 (time 1788821525.15 total 50.04 last 97.80) - create 6617 (time 1788821535.16 total 60.04 last 119.79) - create 7708 (time 1788821545.16 total 70.05 last 109.08) - create 8913 (time 1788821555.20 total 80.08 last 120.04) total: 10000 create in 89.49 seconds: 111.74 ops/second PASS 37 (101s) == sanityn test 39a: file mtime does not change after rename ========================================================== 18:52:52 (1788821572) repeat after cancel_lru_locks PASS 39a (10s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 18:53:03 (1788821583) repeat after cancel_lru_locks PASS 39b (7s) == sanityn test 39c: check truncate mtime update ================================================================================ 18:53:10 (1788821590) repeat after cancel_lru_locks PASS 39c (7s) == sanityn test 39d: sync write should update mtime ====== 18:53:17 (1788821597) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 18:53:25 (1788821605) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (17s) == sanityn test 40c: pdirops: link and others ======================================================================== 18:53:42 (1788821622) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (18s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 18:54:01 (1788821641) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (16s) == sanityn test 40e: pdirops: rename and others ======================================================================== 18:54:16 (1788821656) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (16s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 18:54:32 (1788821672) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (13s) == sanityn test 41b: pdirops: create vs create ======================================================================== 18:54:46 (1788821686) open: File exists Conflict PASS 41b (12s) == sanityn test 41c: pdirops: create vs link ======================================================================== 18:54:58 (1788821698) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (10s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 18:55:09 (1788821709) Conflict PASS 41d (11s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 18:55:20 (1788821720) Conflict PASS 41e (11s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 18:55:31 (1788821731) Conflict PASS 41f (11s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 18:55:42 (1788821742) Conflict PASS 41g (11s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 18:55:53 (1788821753) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (12s) == sanityn test 41i: reint_open: create vs create ======== 18:56:05 (1788821765) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1031s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 19:13:16 (1788822796) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (12s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 19:13:29 (1788822809) open: File exists Conflict PASS 42b (11s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 19:13:40 (1788822820) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (13s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 19:13:53 (1788822833) Conflict PASS 42d (15s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 19:14:08 (1788822848) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (11s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 19:14:19 (1788822859) Conflict PASS 42f (13s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 19:14:32 (1788822872) Conflict PASS 42g (13s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 19:14:45 (1788822885) Conflict PASS 42h (13s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 19:14:58 (1788822898) PASS 43a (63s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 19:16:01 (1788822961) Conflict PASS 43b (12s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 19:16:13 (1788822973) Conflict PASS 43c (12s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 19:16:25 (1788822985) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (11s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 19:16:37 (1788822997) Conflict PASS 43e (10s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 19:16:47 (1788823007) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (13s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 19:17:00 (1788823020) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (17s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 19:17:17 (1788823037) Conflict PASS 43h (14s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 19:17:31 (1788823051) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 19:17:34 (1788823054) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (113s) == sanityn test 43k: unlink vs create ==================== 19:19:27 (1788823167) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1208s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 19:39:35 (1788824375) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Mon Sep 7 19:39:44 EDT 2026 PASS 44a (14s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 19:39:49 (1788824389) open: File exists Conflict PASS 44b (14s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 19:40:03 (1788824403) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (14s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 19:40:17 (1788824417) Conflict PASS 44d (14s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 19:40:31 (1788824431) Conflict PASS 44e (14s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 19:40:45 (1788824445) Conflict PASS 44f (13s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 19:40:58 (1788824458) Conflict PASS 44g (12s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 19:41:11 (1788824471) Conflict PASS 44h (14s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 19:41:24 (1788824484) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (3s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 19:41:27 (1788824487) PASS 45a (117s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 19:43:24 (1788824604) Conflict PASS 45b (13s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 19:43:37 (1788824617) Conflict PASS 45c (12s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 19:43:49 (1788824629) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (11s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 19:44:00 (1788824640) Conflict PASS 45e (11s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 19:44:12 (1788824652) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (11s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 19:44:23 (1788824663) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (10s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 19:44:34 (1788824674) Conflict PASS 45h (11s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 19:44:45 (1788824685) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (2s) == sanityn test 45j: read vs rename ====================== 19:44:48 (1788824688) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (794s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 19:58:02 (1788825482) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (8s) == sanityn test 46b: pdirops: link vs create ======================================================================== 19:58:10 (1788825490) open: File exists Conflict PASS 46b (8s) == sanityn test 46c: pdirops: link vs link ======================================================================== 19:58:18 (1788825498) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (8s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 19:58:26 (1788825506) Conflict PASS 46d (8s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 19:58:34 (1788825514) Conflict PASS 46e (8s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 19:58:42 (1788825522) Conflict PASS 46f (8s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 19:58:51 (1788825531) Conflict PASS 46g (8s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 19:58:58 (1788825538) Conflict PASS 46h (8s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 19:59:06 (1788825546) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (2s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 19:59:08 (1788825548) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (3s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 19:59:11 (1788825551) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 19:59:13 (1788825553) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (2s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 19:59:15 (1788825555) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (2s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 19:59:17 (1788825557) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (2s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 19:59:19 (1788825559) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 19:59:21 (1788825561) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (2s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 19:59:22 (1788825562) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0755749 s, 135 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (9s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 19:59:31 (1788825571) 0+1 records in 0+1 records out 158 bytes copied, 0.00865026 s, 18.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218578) - No such process PASS 51a (5s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 19:59:36 (1788825576) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00691844 s, 148 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323521 s, 32.4 MB/s 3145728 PASS 51b (16s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 19:59:52 (1788825592) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0166271 s, 61.6 kB/s PASS 51c (9s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 20:00:01 (1788825601) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195701 s, 53.6 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (4s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 20:00:05 (1788825605) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 read lease(1) has applied. PASS 51e (6s) == sanityn test 54: rename locking ======================= 20:00:11 (1788825611) llite.lustre-ffff9131c3268800.xattr_cache=1 llite.lustre-ffff9131e025f800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:00:12.000000000 -0400 Modify: 2026-09-07 20:00:12.000000000 -0400 Change: 2026-09-07 20:00:12.000000000 -0400 Birth: 2026-09-07 20:00:13.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:00:18.000000000 -0400 Modify: 2026-09-07 20:00:19.000000000 -0400 Change: 2026-09-07 20:00:19.000000000 -0400 Birth: 2026-09-07 20:00:19.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:00:24.000000000 -0400 Modify: 2026-09-07 20:00:24.000000000 -0400 Change: 2026-09-07 20:00:24.000000000 -0400 Birth: 2026-09-07 20:00:25.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:00:24.000000000 -0400 Modify: 2026-09-07 20:00:24.000000000 -0400 Change: 2026-09-07 20:00:24.000000000 -0400 Birth: 2026-09-07 20:00:25.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:00:30.000000000 -0400 Modify: 2026-09-07 20:00:30.000000000 -0400 Change: 2026-09-07 20:00:30.000000000 -0400 Birth: 2026-09-07 20:00:31.000000000 -0400 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:00:30.000000000 -0400 Modify: 2026-09-07 20:00:30.000000000 -0400 Change: 2026-09-07 20:00:30.000000000 -0400 Birth: 2026-09-07 20:00:31.000000000 -0400 PASS 54 (27s) == sanityn test 55a: rename vs unlink target dir ========= 20:00:38 (1788825638) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (8s) == sanityn test 55b: rename vs unlink source dir ========= 20:00:46 (1788825646) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (9s) == sanityn test 55c: rename vs unlink orphan target dir == 20:00:55 (1788825655) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (14s) == sanityn test 55d: rename file vs link ================= 20:01:09 (1788825669) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (11s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 20:01:20 (1788825680) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (2s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 20:01:22 (1788825682) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (14s) == sanityn test 55g: rename: race with trylock =========== 20:01:36 (1788825696) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (17s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 20:01:53 (1788825713) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788825713 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 1.03456 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (62s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 20:02:55 (1788825775) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788825777 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 2.77441 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (51s) == sanityn test 60: Verify data_version behaviour ======== 20:03:46 (1788825826) PASS 60 (4s) == sanityn test 70a: cd directory && rm directory ======== 20:03:51 (1788825831) PASS 70a (4s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 20:03:54 (1788825834) PASS 70b (4s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 20:03:58 (1788825838) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00556144 s, 11.8 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 20:04:00 (1788825840) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00896209 s, 7.3 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (2s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 20:04:02 (1788825842) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 20:04:03 (1788825843) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00813013 s, 8.1 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (2s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 20:04:05 (1788825845) llite.lustre-ffff9131c3268800.xattr_cache=1 llite.lustre-ffff9131e025f800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (3s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 20:04:08 (1788825848) llite.lustre-ffff9131c3268800.xattr_cache=1 llite.lustre-ffff9131e025f800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (4s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 20:04:12 (1788825852) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00891934 s, 115 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00928759 s, 110 kB/s 231368: get lock2 231369: get lock1 231368: done 231368: get lock1 231369: done 231369: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231369: cannot get lock2: Resource deadlock avoided 231369: exit rc=0 231368: done 231368: exit rc=0 PASS 74 (7s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 20:04:19 (1788825859) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0211513 s, 99.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229601 s, 45.7 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 231949 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (9s) == sanityn test 76: Verify MDT open_files listing ======== 20:04:28 (1788825868) opening files: ........ closing files: ........ PASS 76 (28s) == sanityn test 77a: check FIFO NRS policy =============== 20:04:56 (1788825896) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0952767 s, 176 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018738 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020642 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528654 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569339 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311252 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529835 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633781 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480164 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618432 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661362 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824974 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892386 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866069 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648589 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836215 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843868 s, 12.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9131c3268800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9131c3268800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107288 s, 97.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197392 s, 53.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356131 s, 29.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00780968 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174573 s, 60.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00685002 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00473589 s, 221 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00480607 s, 218 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328973 s, 31.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304511 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030628 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417377 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00738352 s, 142 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335985 s, 31.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127977 s, 81.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00998296 s, 105 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (4s) == sanityn test 77b: check CRR-N NRS policy ============== 20:05:00 (1788825900) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.103904 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183987 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365976 s, 28.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312885 s, 33.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485083 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477493 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545268 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433399 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699051 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823348 s, 12.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9131c3268800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9131c3268800.lock_unused_count=5 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744107 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728013 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739508 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639499 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749627 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773156 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704719 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204912 s, 51.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271528 s, 38.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214743 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307856 s, 34.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321654 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402418 s, 26.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304498 s, 34.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402413 s, 26.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381669 s, 27.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263988 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031401 s, 33.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306287 s, 34.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257075 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266503 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031859 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035821 s, 29.3 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg303-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg303-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0875136 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381494 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473203 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634868 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651996 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445454 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378749 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821193 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494002 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643851 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464295 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609096 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729201 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743527 s, 14.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104511 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731243 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106423 s, 9.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9131c3268800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9131c3268800.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271982 s, 38.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026426 s, 39.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137055 s, 76.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291821 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109801 s, 95.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346151 s, 30.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204255 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325077 s, 32.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234265 s, 44.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219453 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00401386 s, 261 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00665304 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033467 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00337293 s, 311 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00954558 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0030487 s, 344 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (5s) == sanityn test 77c: check ORR NRS policy ================ 20:05:05 (1788825905) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.155659 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151257 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233919 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383291 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043277 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039772 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393932 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271522 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503984 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267047 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531907 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047405 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594998 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632083 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496657 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574072 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537733 s, 19.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9131c3268800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9131c3268800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165346 s, 63.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203078 s, 51.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252556 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021374 s, 49.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328893 s, 31.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259198 s, 40.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256078 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109842 s, 95.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029539 s, 35.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00560097 s, 187 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197183 s, 53.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026939 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102009 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283853 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00330013 s, 318 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9131c3268800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00278991 s, 376 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0878761 s, 191 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161797 s, 64.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206163 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269139 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427129 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042119 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334689 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265625 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456861 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478424 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441885 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436188 s, 24.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9131c3268800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9131c3268800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758922 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644483 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679182 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727439 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558367 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244566 s, 42.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021433 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213829 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217444 s, 48.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291606 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267684 s, 39.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243241 s, 43.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9131c3268800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227055 s, 46.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229608 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229845 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00396643 s, 264 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00417019 s, 251 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222471 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01217 s, 86.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00848508 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100301 s, 105 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0749416 s, 224 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110017 s, 95.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313534 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362107 s, 29.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654816 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701469 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521453 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567439 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052144 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602927 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539367 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723261 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388972 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759204 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703032 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660205 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104468 s, 10.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9131c3268800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9131c3268800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174713 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244359 s, 42.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185957 s, 56.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235479 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240853 s, 43.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200083 s, 52.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297056 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258573 s, 40.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224276 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315567 s, 33.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028242 s, 37.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00673644 s, 156 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00971358 s, 108 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00434746 s, 241 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229227 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249422 s, 42.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (6s) == sanityn test 77d: check TRR nrs policy ================ 20:05:11 (1788825911) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0762643 s, 220 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151221 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460418 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494091 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516391 s, 20.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543361 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418363 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239809 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375368 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454794 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307668 s, 34.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589345 s, 17.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573021 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511664 s, 20.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544242 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694545 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720303 s, 14.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9131c3268800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9131c3268800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166327 s, 63.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220136 s, 47.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202561 s, 51.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297363 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013474 s, 77.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272769 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168566 s, 62.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219477 s, 47.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0043402 s, 242 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256359 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200378 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120236 s, 87.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112488 s, 93.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283146 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031879 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307644 s, 34.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg303-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg303-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0867064 s, 193 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133215 s, 78.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028315 s, 37.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266947 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384889 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450438 s, 23.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036205 s, 29.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411977 s, 25.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471492 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332807 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394934 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381324 s, 27.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465758 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194196 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375042 s, 28.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9131c3268800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642513 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505071 s, 20.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118774 s, 88.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243708 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152488 s, 68.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293862 s, 35.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173978 s, 60.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174851 s, 60.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026507 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209582 s, 50.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216973 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286535 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206982 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022455 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245513 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226328 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023623 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258608 s, 40.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.105247 s, 159 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138468 s, 75.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128689 s, 81.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309285 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426924 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480153 s, 21.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359733 s, 29.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405715 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528466 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501715 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337381 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415941 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388793 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048221 s, 21.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473963 s, 22.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461833 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580613 s, 18.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9131c3268800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9131c3268800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232199 s, 45.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220952 s, 47.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194817 s, 53.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212179 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224092 s, 46.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143074 s, 73.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271563 s, 38.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147718 s, 71.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293727 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200385 s, 52.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247273 s, 42.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189726 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272889 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251455 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00498933 s, 210 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9131c3268800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113485 s, 92.4 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (6s) == sanityn test 77e: check TBF NID nrs policy ============ 20:05:17 (1788825917) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.3@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.103875 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239669 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247457 s, 42.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426704 s, 24.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508796 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306899 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481052 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639608 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460668 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588397 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042141 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568367 s, 18.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773136 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674646 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775332 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493161 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060021 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026352 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179671 s, 58.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0034191 s, 307 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543179 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662067 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060907 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491355 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428134 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073649 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322805 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298708 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275145 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226242 s, 46.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196769 s, 53.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127853 s, 82.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00943534 s, 111 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.117249 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217806 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173731 s, 60.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380206 s, 27.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215768 s, 48.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469437 s, 22.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608685 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629813 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387853 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651797 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375411 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387382 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481791 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067888 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389386 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594243 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562708 s, 18.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124588 s, 84.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183679 s, 57.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026977 s, 38.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0041007 s, 256 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00404207 s, 259 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00437257 s, 240 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00571203 s, 184 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219366 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248537 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00273522 s, 383 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315175 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314762 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446944 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143653 s, 73.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00907437 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00490531 s, 214 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg303-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg303-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.144693 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266307 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340384 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317207 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286826 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406364 s, 25.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471487 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592694 s, 17.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857601 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618823 s, 16.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606296 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557886 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947377 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914075 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721672 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083607 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913059 s, 11.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9131c3268800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320743 s, 32.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03725 s, 28.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360109 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198321 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036443 s, 28.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333308 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297099 s, 35.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265879 s, 39.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215115 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419436 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345535 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344451 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032775 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309194 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268574 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029565 s, 35.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg303-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg303-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0872131 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102168 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192595 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285943 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135103 s, 77.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376688 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127137 s, 82.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240154 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558941 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393698 s, 26.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468946 s, 22.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561541 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552619 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556734 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536251 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356296 s, 29.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9131c3268800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9131c3268800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181491 s, 57.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9131c3268800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017722 s, 59.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218954 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00356807 s, 294 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00371 s, 283 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292153 s, 35.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222465 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319961 s, 32.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270064 s, 38.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173268 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205371 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00373501 s, 281 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272727 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00578261 s, 181 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293027 s, 35.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311318 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00291753 s, 359 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg303-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg303-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (11s) == sanityn test 77f: check TBF JobID nrs policy ========== 20:05:28 (1788825928) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg303-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.108678 s, 154 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313776 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038087 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593839 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806845 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647465 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781935 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440661 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833935 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748265 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674011 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604668 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605221 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683064 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747696 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677027 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061464 s, 17.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192854 s, 54.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203746 s, 51.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250294 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310168 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025058 s, 41.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204312 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325357 s, 32.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192101 s, 54.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207841 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247771 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275405 s, 38.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290961 s, 36.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262995 s, 39.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264145 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321537 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00251804 s, 416 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg303-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg303-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg303-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0879332 s, 191 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147967 s, 70.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205101 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271274 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457214 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539475 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329584 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556304 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515141 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529125 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595402 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482097 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478418 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482653 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467471 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498005 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068252 s, 15.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154177 s, 68.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182248 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155847 s, 67.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169704 s, 61.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213317 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187182 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262679 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190461 s, 55.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312014 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289026 s, 36.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256012 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259611 s, 40.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271965 s, 38.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027715 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274032 s, 38.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284104 s, 36.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg303-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg303-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg303-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0860784 s, 195 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151223 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243727 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443195 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389365 s, 26.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318985 s, 32.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296763 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608349 s, 17.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618444 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00892039 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573387 s, 18.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559486 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500652 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213324 s, 49.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9131c3268800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9131c3268800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299306 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982338 s, 10.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983849 s, 10.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9131c3268800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151289 s, 69.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023891 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201742 s, 52.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141849 s, 73.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145299 s, 72.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401641 s, 26.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018079 s, 58.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177774 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029361 s, 35.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196014 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198412 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220516 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256407 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266171 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272598 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277705 s, 37.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg303-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.125996 s, 133 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167759 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219874 s, 47.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378999 s, 27.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252711 s, 41.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500215 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549111 s, 19.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565486 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523988 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371235 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556736 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579404 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617315 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743303 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636586 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569801 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597363 s, 17.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9131c3268800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9131c3268800.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106126 s, 98.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015477 s, 67.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208382 s, 50.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153224 s, 68.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292847 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022677 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188501 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254739 s, 41.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207557 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020766 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213321 s, 49.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257441 s, 40.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266671 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318493 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026911 s, 39.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278325 s, 37.7 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg303-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg303-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (9s) == sanityn test 77g: Change TBF type directly ============ 20:05:37 (1788825937) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (5s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 20:05:42 (1788825942) oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg303-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg303-client: oleg303-server: ssh exited with exit code 2 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg303-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg303-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg303-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 PASS 77h (6s) == sanityn test 77i: Change rank of TBF rule ============= 20:05:48 (1788825948) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (11s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 20:05:59 (1788825959) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.262538 s, 63.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215738 s, 48.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025337 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137643 s, 76.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203884 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252207 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389966 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00824493 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265862 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040332 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640696 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595036 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669931 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256981 s, 4.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238885 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286215 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266526 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209807 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270145 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323542 s, 32.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301286 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233914 s, 44.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319854 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389454 s, 26.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311873 s, 33.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346556 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355884 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302072 s, 34.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389589 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304259 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345443 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380105 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422944 s, 24.8 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.06534 s, 20.7 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4647 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (54s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 20:06:54 (1788826014) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg303-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg303-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg303-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg303-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg303-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23363 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255649 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323243 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03362 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235719 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398413 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430941 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399164 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156757 s, 66.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15474 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155827 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162743 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157655 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155841 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179129 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181156 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196331 s, 5.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392447 s, 26.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346925 s, 30.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404801 s, 25.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182028 s, 57.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045229 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471714 s, 22.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495784 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527932 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337956 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527334 s, 19.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350042 s, 30.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344115 s, 30.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375247 s, 27.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367423 s, 28.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516182 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418632 s, 25.0 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.805 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9102 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg303-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg303-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg303-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg303-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg303-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22455 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283677 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256792 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418409 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154509 s, 67.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342111 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467732 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323469 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302141 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152337 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15355 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170574 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12121 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148663 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143542 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167093 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202724 s, 5.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263172 s, 39.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301195 s, 34.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444284 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449234 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426194 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389272 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505848 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342701 s, 30.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539343 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565537 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352539 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369102 s, 28.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420859 s, 24.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370049 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451084 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410955 s, 25.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7949 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9414 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (138s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 20:09:12 (1788826152) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg303-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg303-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg303-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg303-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg303-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23303 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262336 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295222 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019444 s, 53.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410042 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160115 s, 65.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448639 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366236 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262022 s, 40.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12956 s, 8.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135506 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142261 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158424 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171308 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170588 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129251 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147857 s, 7.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326652 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245599 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320707 s, 32.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034074 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356497 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378394 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353975 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274652 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410604 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397492 s, 26.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436515 s, 24.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331646 s, 31.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033066 s, 31.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459747 s, 22.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426601 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381038 s, 27.5 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7861 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9718 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg303-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg303-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg303-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg303-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg303-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27158 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039572 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697621 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883368 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094389 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151551 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161257 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159616 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172757 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159767 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188899 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200173 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200755 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221198 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216269 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225799 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258162 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12964 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135765 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124103 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131533 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110414 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166979 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111297 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11751 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109136 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159542 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182472 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126924 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186434 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161671 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168139 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16808 s, 6.2 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg303-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg303-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.579 s, 5.1 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.5565 s, 5.1 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (148s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 20:11:40 (1788826300) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg303-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.427839 s, 39.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469678 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657389 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038707 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459154 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129703 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173047 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192809 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155789 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169449 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206113 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208597 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2112 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247656 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276035 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269714 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288549 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648783 s, 16.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048801 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103616 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587956 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580041 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802641 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646664 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747479 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798107 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824447 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868258 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880674 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113469 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10901 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109078 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115972 s, 9.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.2849 s, 8.5 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99908 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.3@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.3@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.29317 s, 57.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152852 s, 68.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237445 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454553 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246172 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789214 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903174 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797257 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833831 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606821 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955691 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24391 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241754 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247203 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26272 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306536 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310923 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270654 s, 38.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271812 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052772 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376389 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393751 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356132 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529234 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409749 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421119 s, 24.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627243 s, 16.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540724 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049646 s, 21.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455057 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011746 s, 89.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020279 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679658 s, 15.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg303-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg303-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.05371 s, 14.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94156 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.3@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg303-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.326721 s, 51.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524414 s, 20.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198041 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624528 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501938 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705045 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526749 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102296 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362328 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110036 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825606 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433333 s, 24.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194563 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138783 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202334 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200935 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140295 s, 7.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668093 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189223 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566924 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626968 s, 16.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820927 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638558 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597966 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613614 s, 17.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062588 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732005 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834489 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814092 s, 12.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588773 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533113 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846583 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882068 s, 11.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg303-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg303-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.38585 s, 16.4 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.05378 s, 20.7 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg303-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.484124 s, 34.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336143 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00801792 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248042 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420906 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403372 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109224 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134817 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125107 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133966 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127963 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139935 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183683 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163313 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157242 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196282 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225541 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600164 s, 17.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00927188 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233338 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111628 s, 93.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544936 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677378 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632522 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691436 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798771 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059978 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060655 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00937034 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00598669 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132324 s, 79.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098631 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993309 s, 10.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4385 s, 10.0 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91046 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.06084 s, 14.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93064 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg303-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.00216 s, 16.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647548 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429896 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580296 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130465 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103267 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144395 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14011 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174008 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188267 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135512 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183855 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195309 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184248 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201598 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305904 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511921 s, 20.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504126 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06506 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419088 s, 25.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463348 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430448 s, 24.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500628 s, 20.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050447 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452653 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474292 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504961 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510455 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527832 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575395 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628604 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616482 s, 17.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9475 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6284 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg303-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.427922 s, 39.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205122 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231375 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440543 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466993 s, 22.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103766 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967805 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130359 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126179 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993365 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130419 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145469 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1402 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142117 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167079 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168552 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191775 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204512 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692644 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263959 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310259 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910906 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892467 s, 11.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522781 s, 20.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509405 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550278 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557413 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400959 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707304 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068127 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693005 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911849 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151624 s, 6.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg303-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg303-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.85129 s, 13.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9261 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg303-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.515966 s, 32.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221901 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148764 s, 70.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324801 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367342 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438058 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404031 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679513 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696314 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618489 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560133 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120125 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111891 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156673 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135445 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186976 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210816 s, 5.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258176 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642569 s, 16.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606946 s, 17.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325994 s, 32.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355098 s, 29.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057606 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378695 s, 27.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640087 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692711 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779484 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588311 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487133 s, 21.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496532 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509007 s, 20.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866447 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574157 s, 18.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.123 s, 10.4 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91473 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1442 s, 10.3 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86531 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo PASS 77k (432s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 20:18:52 (1788826732) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg303-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.369229 s, 45.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402614 s, 26.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311571 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184623 s, 56.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228923 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612299 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989026 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104449 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854633 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108138 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108144 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159693 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114796 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152644 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157943 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137591 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186683 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260176 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209813 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211219 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510622 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267006 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443954 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034856 s, 30.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381557 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731338 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413093 s, 25.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557758 s, 18.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570372 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579397 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588203 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582396 s, 18.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615983 s, 17.0 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg303-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg303-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.28138 s, 14.4 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89469 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kb (55s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 20:19:47 (1788826787) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.3@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.3@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.338931 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026831 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548682 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674406 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119588 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147504 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150851 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169579 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977591 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122185 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972975 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124668 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162806 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186403 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129615 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202248 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211518 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489241 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389798 s, 26.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416371 s, 25.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306472 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637814 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585448 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618727 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428074 s, 24.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474386 s, 22.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598803 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664647 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00921207 s, 114 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614467 s, 17.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479012 s, 21.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768846 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864039 s, 12.1 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg303-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg303-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.76435 s, 15.5 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90167 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kc (56s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 20:20:43 (1788826843) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.3@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg303-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.374303 s, 44.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282322 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463429 s, 22.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286175 s, 36.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468391 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561425 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483937 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491633 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421135 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179491 s, 58.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034814 s, 30.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450438 s, 23.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528619 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241729 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017574 s, 59.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932197 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184854 s, 56.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384572 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340113 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259753 s, 40.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367559 s, 28.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255164 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485233 s, 21.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027697 s, 37.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380453 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294748 s, 35.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289128 s, 36.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407923 s, 25.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429633 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332021 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359245 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356841 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395707 s, 26.5 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg303-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg303-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.9029 s, 17.8 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.05048 s, 20.8 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_policies=fifo PASS 77kd (44s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 20:21:27 (1788826887) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg303-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.498751 s, 33.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026804 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392016 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613745 s, 17.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722559 s, 14.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950711 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779835 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831359 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099984 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108064 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11065 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108634 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113254 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166508 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172721 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245892 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251554 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532197 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311748 s, 33.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00994852 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042871 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319491 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439983 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446731 s, 23.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537992 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296429 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925706 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00565972 s, 185 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039616 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543999 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708327 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734278 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576936 s, 18.2 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93165 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86382 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.00499 s, 17.5 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85744 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77ke (98s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 20:23:05 (1788826985) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg303-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.911893 s, 18.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155534 s, 67.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328404 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297184 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352055 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676163 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769402 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553829 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476752 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141593 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155818 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177635 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169399 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194555 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204763 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179968 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19482 s, 5.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450458 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583357 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595659 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533757 s, 19.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567319 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599428 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595635 s, 17.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608102 s, 17.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647631 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654512 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638541 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615765 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667236 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620653 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683294 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065193 s, 16.1 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.671 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9185 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_policies=fifo PASS 77kf (73s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 20:24:18 (1788827058) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg303-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.301683 s, 55.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222969 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211731 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470177 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024554 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503451 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361437 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401621 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604897 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614337 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504837 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576082 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875644 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835364 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567848 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070017 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107089 s, 9.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218016 s, 48.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281723 s, 37.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418665 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472087 s, 22.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388882 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302842 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448493 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348628 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537037 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030347 s, 34.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403836 s, 26.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036508 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433724 s, 24.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038008 s, 27.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449142 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478017 s, 21.9 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg303-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg303-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.11559 s, 20.5 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79703 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg303-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.451464 s, 37.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026226 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010191 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514946 s, 20.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557111 s, 18.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560996 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481145 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054057 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783725 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071905 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880216 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744312 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739243 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195249 s, 5.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197264 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29546 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280539 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024821 s, 42.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526742 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272152 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218967 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204938 s, 51.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217506 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277199 s, 37.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506767 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295291 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272816 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371777 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352878 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338254 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351327 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354388 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362352 s, 28.9 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92883 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7929 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90483 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84336 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77kg (136s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 20:26:34 (1788827194) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg303-client.virtnet /mnt/lustre (opts:) Stopping client oleg303-client.virtnet /mnt/lustre opts: Stopping clients: oleg303-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg303-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg303-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg303-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg303-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg303-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.3,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg303-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.3,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg303-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.3,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69792/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.3,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg303-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg303-server' oleg303-server: oleg303-server.virtnet: executing load_modules_local oleg303-server: kmemleak disabled oleg303-server: Loading modules from /home/green/git/lustre-release/lustre oleg303-server: detected 4 online CPUs by sysfs oleg303-server: MODOPTS_LIBCFS= oleg303-server: Force libcfs to create 2 CPU partitions oleg303-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre Starting client oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre Started clients oleg303-client.virtnet: 192.168.203.103@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre2 Starting client oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre2 Started clients oleg303-client.virtnet: 192.168.203.103@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9131c53d8800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9131c9d52000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9131c53d8800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9131c9d52000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.336562 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363061 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399943 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14412 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151001 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137406 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135725 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155022 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162777 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140017 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134271 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150277 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138163 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210929 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217722 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248312 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245948 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338665 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137077 s, 76.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00407512 s, 257 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212488 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273978 s, 38.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227625 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00372035 s, 282 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280243 s, 37.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220271 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292482 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024437 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00741717 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00815325 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00282305 s, 371 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388184 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509243 s, 20.6 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.12489 s, 20.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99559 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.322993 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306571 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282848 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110331 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116889 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105582 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100027 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120807 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107674 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115165 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127779 s, 8.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122472 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128978 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227978 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20856 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281763 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277033 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351768 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331016 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307796 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207255 s, 50.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206519 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248946 s, 42.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226276 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243826 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242621 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327955 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249349 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228225 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265507 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299389 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262137 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315936 s, 33.2 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.10505 s, 20.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81454 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (149s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 20:29:03 (1788827343) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg303-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (10s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 20:29:13 (1788827353) On MGS 192.168.203.103, active = nodemap.active=1 On MGS 192.168.203.103, TBF.id = nodemap.TBF.id=1 ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.3134 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589876 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416858 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672382 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658978 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211069 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545761 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634307 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101267 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115791 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076414 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128244 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120312 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215934 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225203 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2448 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292438 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320704 s, 32.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181307 s, 57.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252505 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193153 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021583 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186345 s, 56.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223905 s, 46.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236895 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271456 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257071 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286857 s, 36.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025236 s, 41.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265055 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282269 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322755 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317571 s, 33.0 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg303-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg303-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.21824 s, 20.1 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98465 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.203.103, TBF.id = On MGS 192.168.203.103, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.20278 s, 82.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236159 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451578 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882847 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080848 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917103 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917779 s, 11.4 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff9131c53d8800.lock_unused_count=3 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762916 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980166 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863195 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137902 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150751 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00615937 s, 170 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00583753 s, 180 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025745 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156302 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232627 s, 45.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370168 s, 28.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00973039 s, 108 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.006537 s, 160 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166005 s, 63.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184039 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00925386 s, 113 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246452 s, 42.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9131c53d8800.lock_unused_count=9 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197437 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174617 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00312828 s, 335 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00664386 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590351 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630073 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00330515 s, 317 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00336902 s, 311 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188677 s, 5.6 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.27759 s, 32.0 MB/s Write runtime is 4 s, speed is 25.000000 IOPS write rate (25.000000) should exceed limit (20 * 1) Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.12675 s, 93.1 MB/s Read runtime is 2 s, speed is 50.000000 IOPS read rate (50.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg303-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg303-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 On MGS 192.168.203.103, active = nodemap.active=1 On MGS 192.168.203.103, TBF.id = nodemap.TBF.id=2 ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.309635 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418881 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298869 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481774 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427001 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423805 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096913 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954909 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108572 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182013 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173703 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181229 s, 5.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187942 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217357 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200628 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279622 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278158 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192162 s, 54.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275888 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143259 s, 73.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332697 s, 31.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188989 s, 55.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333171 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036393 s, 28.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277048 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276431 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286127 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407661 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233406 s, 44.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281513 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243539 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301141 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037967 s, 27.6 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg303-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg303-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.19446 s, 20.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81389 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start nmdyn nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.331459 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366813 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052073 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651972 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101929 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869474 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128129 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770139 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859808 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858028 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123885 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19518 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201721 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211051 s, 5.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21156 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294925 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261449 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116873 s, 89.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219252 s, 47.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345325 s, 30.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412286 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023574 s, 44.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252629 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330128 s, 31.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260995 s, 40.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284015 s, 36.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337908 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394133 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393162 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484917 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482578 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408667 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452686 s, 23.2 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.23162 s, 20.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97765 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nmdyn ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.203.103, TBF.id = On MGS 192.168.203.103, active = nodemap.active=0 Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.203.103, active = nodemap.active=0 PASS 77kj (182s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 20:32:15 (1788827535) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (5s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 20:32:20 (1788827540) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg303-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 4.04157 s, 259 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg303-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0317 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (53s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 20:33:13 (1788827593) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.19214 s, 87.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399569 s, 26.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345445 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784832 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851245 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752154 s, 13.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9131c53d8800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9131c53d8800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11242 s, 9.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106741 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12826 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130594 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142351 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14378 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142583 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153978 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139625 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150081 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151114 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310913 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278644 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367194 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299917 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283001 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367327 s, 28.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03297 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312477 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242855 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268931 s, 39.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295581 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305488 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304549 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029082 s, 36.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9131c53d8800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9131c53d8800.lock_unused_count=8 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330298 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00265065 s, 396 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg303-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg303-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.04277 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97245 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.369453 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331339 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403026 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766876 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646883 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108051 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990943 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13282 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143316 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10603 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149063 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113863 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131156 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179555 s, 5.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14139 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216663 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21003 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207664 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269041 s, 39.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420404 s, 24.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435584 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038858 s, 27.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341584 s, 30.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413361 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365935 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436379 s, 24.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043882 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429877 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447738 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466375 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488584 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495882 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059315 s, 17.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.27844 s, 19.9 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99058 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (72s) == sanityn test 77o: Changing rank should not panic ====== 20:34:25 (1788827665) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (6s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 20:34:31 (1788827671) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (138s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 20:36:50 (1788827810) mds.MDS.mdt.nrs_policies=tbf oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg303-client: oleg303-server: ssh exited with exit code 36 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg303-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (32s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 20:37:22 (1788827842) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 277668 kill md_thread xargs PASS 77r (53s) == sanityn test 77s: Check TBF LRU shrinker ============== 20:38:15 (1788827895) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, type=jobid, ref 10 touch jobid={touch.*} 1000, type=jobid, ref 0 unlink jobid={unlink.*} 1000, type=jobid, ref 10 unlink jobid={unlink.*} 1000, type=jobid, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (57s) == sanityn test 77ta: Testing for dynamic|static classify parameter setting for TBF ========================================================== 20:39:13 (1788827953) ost.OSS.ost_io.nrs_policies=fifo oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid+uid classify=foo: Invalid argument oleg303-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid+uid classify=foo': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid+uid foo=static: Invalid argument oleg303-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid+uid foo=static': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid+uid classify=: Invalid argument oleg303-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid+uid classify=': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic ost.OSS.ost_io.nrs_policies=tbf nid classify=static ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf reg nid classify=dynamic ost.OSS.ost_io.nrs_policies=tbf hp nid ost.OSS.ost_io.nrs_tbf_rule=start mixed1 nid={0@lo} rate=100 ost.OSS.ost_io.nrs_tbf_rule=stop mixed1 ost.OSS.ost_io.nrs_policies=tbf reg nid classify=static ost.OSS.ost_io.nrs_policies=tbf hp nid classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start mixed2 nid={0@lo} rate=100 ost.OSS.ost_io.nrs_tbf_rule=stop mixed2 ost.OSS.ost_io.nrs_policies=tbf reg nid+uid ost.OSS.ost_io.nrs_policies=tbf hp nid oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start mixed3 nid={0@lo}&uid={500} rate=100: Invalid argument oleg303-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start mixed3 nid={0@lo}&uid={500} rate=100': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=tbf nid classify=static classify=dynamic ost.OSS.ost_io.nrs_policies=tbf reg nid classify=dynamic ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=dynamic state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=static classify=dynamic state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf hp nid classify=static ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf state: stopped fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=static state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_policies=fifo PASS 77ta (39s) == sanityn test 77tB: Request falling to default rule should not panic for dynamic TBF ========================================================== 20:39:51 (1788827991) ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7569 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9415 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77tB.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=start unmatchrule nid={192.0.2.1@tcp} rate=100 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8409 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9262 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77tB.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop unmatchrule ost.OSS.ost_io.nrs_policies=fifo PASS 77tB (134s) == sanityn test 77th: TBF: a mixed classify head pair keeps each head type ========================================================== 20:42:05 (1788828125) ost.OSS.ost_io.nrs_policies=tbf reg nid+opcode ost.OSS.ost_io.nrs_policies=tbf hp nid classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start mixedhp nid={0@lo} rate=100 regular_requests: CPT 0: mixedhp nid={0@lo} 100, type=nid+opcode, ref 0 default * 10000, type=nid+opcode, ref 0 high_priority_requests: CPT 0: mixedhp nid={0@lo} 100, type=nid, ref 0 default * 10000, type=nid, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop mixedhp ost.OSS.ost_io.nrs_policies=fifo PASS 77th (9s) == sanityn test 77tb: Testing for dynamic classify TBF Scheduler ========================================================== 20:42:14 (1788828134) ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=dynamic state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=dynamic state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_tbf_classify= reg_classify:dynamic hp_classify:dynamic ost.OSS.ost_io.nrs_tbf_rule=start nidall nid={0@lo 192.168.203.3@tcp} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.203.3@tcp}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.203.3@tcp}&opcode={ost_write} rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: nidwr nid={0@lo 192.168.203.3@tcp}&opcode={ost_write} 15, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.203.3@tcp}&opcode={ost_read} 5, type=nid+opcode, ref 0 nidall nid={0@lo 192.168.203.3@tcp} 10, type=nid, ref 0 default * 10000, type=nid, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.203.3@tcp}&opcode={ost_write} 15, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.203.3@tcp}&opcode={ost_read} 5, type=nid+opcode, ref 0 nidall nid={0@lo 192.168.203.3@tcp} 10, type=nid, ref 0 default * 10000, type=nid, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.652776 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905517 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837572 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657919 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215048 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205261 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192748 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240205 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206541 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191593 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192453 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272663 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230604 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231383 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24022 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203885 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260768 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547583 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556114 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057083 s, 18.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059139 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109925 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654828 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725105 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758359 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755941 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070638 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879402 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859173 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945617 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903202 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896996 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105612 s, 9.9 MB/s removed '/mnt/lustre/d77tb.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77tb.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77tb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.92038 s, 13.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5658 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77tb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nidall ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_policies=fifo PASS 77tb (69s) == sanityn test 77tc: Testing for the setting of allowed TBF classification type ========================================================== 20:43:23 (1788828203) oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid classify=dynamic allow=nid allow=nid: File exists oleg303-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid classify=dynamic allow=nid allow=nid': File exists pdsh@oleg303-client: oleg303-server: ssh exited with exit code 17 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid allow=nid allow=uid: Invalid argument oleg303-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid allow=nid allow=uid': Invalid argument pdsh@oleg303-client: oleg303-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic allow=nid+opcode allow=nid allow=uid+opcode oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start another uid={500} rate=10: Operation not permitted oleg303-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start another uid={500} rate=10': Operation not permitted pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start another uid={500}&opcode={ost_read}&gid={500} rate=10: Operation not permitted oleg303-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start another uid={500}&opcode={ost_read}&gid={500} rate=10': Operation not permitted pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 ost.OSS.ost_io.nrs_tbf_rule=start nidall nid={0@lo} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo}&opcode={ost_read} rate=5 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nidall ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_policies=fifo oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nodemap classify=dynamic allow=nid allow=uid allow=gid allow=jobid allow=opcode allow=projid allow=nid+uid allow=nid+gid allow=uid+gid allow=nid+jobid allow=uid+jobid allow=gid+jobid allow=nid+opcode allow=uid+opcode allow=gid+opcode allow=nid+projid: Argument list too long oleg303-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nodemap classify=dynamic allow=nid allow=uid allow=gid allow=jobid allow=opcode allow=projid allow=nid+uid allow=nid+gid allow=uid+gid allow=nid+jobid allow=uid+jobid allow=gid+jobid allow=nid+opcode allow=uid+opcode allow=gid+opcode allow=nid+projid': Argument list too long pdsh@oleg303-client: oleg303-server: ssh exited with exit code 7 Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo PASS 77tc (26s) == sanityn test 77td: Testing Number of allowed rules larger or smaller than threshold ========================================================== 20:43:49 (1788828229) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg303-client.virtnet /mnt/lustre (opts:) Stopping client oleg303-client.virtnet /mnt/lustre opts: Stopping clients: oleg303-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg303-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg303-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg303-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg303-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg303-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.3,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg303-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.3,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg303-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.3,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69792/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.3,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg303-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg303-server' oleg303-server: oleg303-server.virtnet: executing load_modules_local oleg303-server: kmemleak disabled oleg303-server: Loading modules from /home/green/git/lustre-release/lustre oleg303-server: detected 4 online CPUs by sysfs oleg303-server: MODOPTS_LIBCFS= oleg303-server: Force libcfs to create 2 CPU partitions oleg303-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre Starting client oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre Started clients oleg303-client.virtnet: 192.168.203.103@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre2 Starting client oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre2 Started clients oleg303-client.virtnet: 192.168.203.103@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9131c5d46000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9131c9d05800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9131c5d46000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9131c9d05800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic allow=projid+opcode allow=projid ost.OSS.ost_io.nrs_tbf_allowed_types= regular_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid high_priority_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid 8 ost.OSS.ost_io.nrs_tbf_rule=start rulerd0 projid={100}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr0 projid={100}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start ruleall0 projid={200} rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall0 projid={200} 15, type=projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, type=opcode+projid, ref 0 default * 10000, type=nid+opcode+projid, ref 0 high_priority_requests: CPT 0: ruleall0 projid={200} 15, type=projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, type=opcode+projid, ref 0 default * 10000, type=nid+opcode+projid, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.357984 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397782 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789444 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849858 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118023 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104838 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07134 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832812 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135771 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135304 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135637 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128003 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22551 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156387 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281057 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378852 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308172 s, 34.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29525 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646892 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416057 s, 25.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155065 s, 67.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542002 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337043 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344896 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129506 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896758 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640768 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411664 s, 25.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9131c9d05800.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635056 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727274 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00890458 s, 118 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00896218 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480076 s, 21.8 MB/s removed '/mnt/lustre/d77td.sanityn/nrs_r_oleg303-client.virtnet' removed '/mnt/lustre/d77td.sanityn/nrs_w_oleg303-client.virtnet' removed directory '/mnt/lustre/d77td.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99187 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5746 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.56341 s, 13.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.62104 s, 15.8 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=change default rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall0 projid={200} 15, type=projid, ref 1 rulewr0 projid={100}&opcode={ost_write} 10, type=opcode+projid, ref 1 rulerd0 projid={100}&opcode={ost_read} 5, type=opcode+projid, ref 1 default * 15, type=nid+opcode+projid, ref 2 high_priority_requests: CPT 0: ruleall0 projid={200} 15, type=projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, type=opcode+projid, ref 0 default * 15, type=nid+opcode+projid, ref 1 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.2485 s, 14.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.62448 s, 15.8 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic allow=projid+opcode allow=projid ost.OSS.ost_io.nrs_tbf_allowed_types= regular_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid high_priority_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid 8 ost.OSS.ost_io.nrs_tbf_rule=start rulerd0 projid={100}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr0 projid={100}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd1 projid={101}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr1 projid={101}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd2 projid={102}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr2 projid={102}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd3 projid={103}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr3 projid={103}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd4 projid={104}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr4 projid={104}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start ruleall0 projid={200} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall1 projid={201} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall2 projid={202} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall3 projid={203} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall4 projid={204} rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall4 projid={204} 15, type=projid, ref 0 ruleall3 projid={203} 15, type=projid, ref 0 ruleall2 projid={202} 15, type=projid, ref 0 ruleall1 projid={201} 15, type=projid, ref 0 ruleall0 projid={200} 15, type=projid, ref 0 rulewr4 projid={104}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, type=opcode+projid, ref 0 default * 10000, type=nid+opcode+projid, ref 0 high_priority_requests: CPT 0: ruleall4 projid={204} 15, type=projid, ref 0 ruleall3 projid={203} 15, type=projid, ref 0 ruleall2 projid={202} 15, type=projid, ref 0 ruleall1 projid={201} 15, type=projid, ref 0 ruleall0 projid={200} 15, type=projid, ref 0 rulewr4 projid={104}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, type=opcode+projid, ref 0 default * 10000, type=nid+opcode+projid, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.468446 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086494 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857503 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112198 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2023 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196665 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23118 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224176 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160099 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246835 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226118 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252759 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223603 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25946 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251107 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281347 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9131c9d05800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9131c9d05800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301151 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00638494 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455002 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00379768 s, 276 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402741 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900535 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414521 s, 25.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685414 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340892 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436081 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00603358 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447328 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476922 s, 22.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492072 s, 21.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484822 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049665 s, 21.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268262 s, 39.1 MB/s removed '/mnt/lustre/d77td.sanityn/nrs_w_oleg303-client.virtnet' removed '/mnt/lustre/d77td.sanityn/nrs_r_oleg303-client.virtnet' removed directory '/mnt/lustre/d77td.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1926 s, 10.3 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5567 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.10577 s, 14.8 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.63107 s, 15.8 MB/s Read runtime is 8 s, speed is 12.500000 IOPS cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=change default rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall4 projid={204} 15, type=projid, ref 0 ruleall3 projid={203} 15, type=projid, ref 0 ruleall2 projid={202} 15, type=projid, ref 0 ruleall1 projid={201} 15, type=projid, ref 0 ruleall0 projid={200} 15, type=projid, ref 1 rulewr4 projid={104}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, type=opcode+projid, ref 1 rulerd0 projid={100}&opcode={ost_read} 5, type=opcode+projid, ref 1 default * 15, type=nid+opcode+projid, ref 2 high_priority_requests: CPT 0: ruleall4 projid={204} 15, type=projid, ref 0 ruleall3 projid={203} 15, type=projid, ref 0 ruleall2 projid={202} 15, type=projid, ref 0 ruleall1 projid={201} 15, type=projid, ref 0 ruleall0 projid={200} 15, type=projid, ref 0 rulewr4 projid={104}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, type=opcode+projid, ref 0 default * 15, type=nid+opcode+projid, ref 1 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.02381 s, 14.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.63864 s, 15.8 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo PASS 77td (353s) == sanityn test 77tf: TBF: rule churn under load on a dynamic classify policy ========================================================== 20:49:42 (1788828582) jobid_name=%e.%p jobid_var=TEST_77TF_JOBVAR ost.OSS.ost_io.nrs_policies=tbf jobid+opcode classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77tf (39s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 20:50:22 (1788828622) oleg303-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg303-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg303-client: oleg303-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (5s) == sanityn test 79: xattr: intent error ================== 20:50:27 (1788828627) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115272381366401 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:50:28.000000000 -0400 Modify: 2026-09-07 20:50:28.000000000 -0400 Change: 2026-09-07 20:50:28.000000000 -0400 Birth: 2026-09-07 20:50:29.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (16s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 20:50:43 (1788828643) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (2s) == sanityn test 80b: Accessing directory during migration ========================================================== 20:50:45 (1788828645) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 20:50:48 (1788828648) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 20:50:51 (1788828651) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (2s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 20:50:54 (1788828654) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 20:50:56 (1788828656) - open/close 824 (time 1788828670.51 total 10.01 last 82.33) - open/close 1716 (time 1788828680.51 total 20.01 last 89.14) total: 2500 open/close in 28.16 seconds: 88.77 ops/second snapshot_time 1788828750.692495544 secs.nsecs start_time 1788828659.776881517 secs.nsecs elapsed_time 90.915614027 secs.nsecs open 2505 samples [usecs] 118 11060 3278216 6282150812 close 2504 samples [usecs] 137 8898 835768 539742262 mknod 2501 samples [usecs] 528 10949 2879113 4985164841 unlink 2500 samples [usecs] 704 46654 4358609 19128310195 mkdir 2 samples [usecs] 1531 1619 3150 4965122 rmdir 2 samples [usecs] 15652 17918 33570 566039828 rename 2500 samples [usecs] 444 20932 10352926 72814281118 getattr 3761 samples [usecs] 7 3664 200326 102663864 parallel_rename_file 2500 samples [usecs] 444 20932 10352926 72814281118 crossdir_rename 2500 samples [usecs] 444 20932 10352926 72814281118 readpage 17 samples [usecs] 4079 31827 223430 4003095578 PASS 81d (97s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 20:52:33 (1788828753) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7631 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7631 PASS 82 (4s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 20:52:38 (1788828758) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 20:52:40 (1788828760) fail_loc=0x8000060b PASS 84 (9s) == sanityn test 85: Lustre API root cache race =========== 20:52:50 (1788828770) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788828771 Finishing test test1 at 1788828773 PASS 85 (6s) == sanityn test 90: open/create and unlink striped directory ========================================================== 20:52:56 (1788828776) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (2s) == sanityn test 91: chmod and unlink striped directory === 20:52:58 (1788828778) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (2s) == sanityn test 92: create remote directory under orphan directory ========================================================== 20:53:00 (1788828780) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (2s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 20:53:02 (1788828782) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5833 0x16c9 0x280000400 0 5892 0x1704 0x240000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5834 0x16ca 0x280000400 0 5893 0x1705 0x240000400 rc1=0 and rc2=0 PASS 93 (13s) == sanityn test 94: signal vs CP callback race =========== 20:53:15 (1788828795) osc.lustre-OST0000-osc-ffff9131c5d46000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9131c9d05800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9131c5d46000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9131c9d05800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.167361 s, 48.9 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.09992 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6890: 307871 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0331623 s, 124 kB/s osc.lustre-OST0000-osc-ffff9131c5d46000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9131c9d05800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9131c5d46000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9131c9d05800.idle_timeout=debug PASS 94 (13s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 20:53:28 (1788828808) llite.lustre-ffff9131c5d46000.fast_read=0 llite.lustre-ffff9131c9d05800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0469801 s, 44.6 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9131c5d46000.fast_read=1 llite.lustre-ffff9131c9d05800.fast_read=1 PASS 95a (16s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 20:53:44 (1788828824) llite.lustre-ffff9131c5d46000.fast_read=0 llite.lustre-ffff9131c9d05800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00496438 s, 2.5 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9131c5d46000.fast_read=1 llite.lustre-ffff9131c9d05800.fast_read=1 PASS 95b (11s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 20:53:55 (1788828835) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (2s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 20:53:57 (1788828837) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00445946 s, 918 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (4s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 20:54:01 (1788828841) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0784341 s, 26.7 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (5s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 20:54:06 (1788828846) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10191 s, 20.6 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (4s) == sanityn test 100e: DoM: read on open and file size ==== 20:54:10 (1788828850) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00422693 s, 2.8 MB/s PASS 100e (5s) == sanityn test 101a: Discard DoM data on unlink ========= 20:54:15 (1788828855) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00517085 s, 792 kB/s PASS 101a (3s) == sanityn test 101b: Discard DoM data on rename ========= 20:54:18 (1788828858) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00374203 s, 1.1 MB/s PASS 101b (4s) == sanityn test 101c: Discard DoM data on close-unlink === 20:54:23 (1788828863) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0103516 s, 396 kB/s PASS 101c (5s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 20:54:28 (1788828868) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: a1 13 00 00 02 00 00 00 62 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x2000013a1:0xa62:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (4s) == sanityn test 103: Test size correctness with lockahead ========================================================== 20:54:33 (1788828873) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788828874 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788828877 Finishing test test23 at 1788828877 PASS 103 (11s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 20:54:43 (1788828883) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (2s) == sanityn test 105: Glimpse and lock cancel race ======== 20:54:45 (1788828885) fail_loc=0x416 PASS 105 (25s) == sanityn test 106a: Verify the btime via statx() ======= 20:55:10 (1788828910) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 20:55:13 (1788828913) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00977071 s, 105 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381368937 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:55:14.000000000 -0400 Modify: 2026-09-07 20:55:14.000000000 -0400 Change: 2026-09-07 20:55:14.000000000 -0400 Birth: 2026-09-07 20:55:15.000000000 -0400 /mnt/lustre/f106b.sanityn 144115272381368937 -rw-r--r-- 1 0 0 1788828915 1788828914 1788828914 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381368937 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:55:14.000000000 -0400 Modify: 2026-09-07 20:55:14.000000000 -0400 Change: 2026-09-07 20:55:14.000000000 -0400 Birth: 2026-09-07 20:55:15.000000000 -0400 1788828914 1024 1 PASS 106b (5s) == sanityn test 106c: Verify statx attributes mask ======= 20:55:18 (1788828918) PASS 106c (4s) == sanityn test 107a: Basic grouplock conflict =========== 20:55:22 (1788828922) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.195471 s, 53.6 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5841 0x16d1 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 First grouplock blocks second one PASS 107a (7s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 20:55:30 (1788828930) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.262249 s, 40.0 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5902 0x170e 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7631 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (11s) == sanityn test 108a: lseek: parallel updates ============ 20:55:41 (1788828941) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00587435 s, 1.7 MB/s llite.lustre-ffff9131c5d46000.hybrid_io=0 llite.lustre-ffff9131c9d05800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.46432 s, 1.9 MB/s PASS 108a (9s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 20:55:51 (1788828951) 192.168.203.103@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg303-client.virtnet /mnt/lustre (opts:) 192.168.203.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg303-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre2 192.168.203.103@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg303-client.virtnet /mnt/lustre (opts:) Stopping client oleg303-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg303-server' oleg303-server: oleg303-server.virtnet: executing load_modules_local oleg303-server: kmemleak disabled oleg303-server: Loading modules from /home/green/git/lustre-release/lustre oleg303-server: detected 4 online CPUs by sysfs oleg303-server: MODOPTS_LIBCFS= oleg303-server: Force libcfs to create 2 CPU partitions oleg303-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre2 192.168.203.103@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg303-client.virtnet /mnt/lustre (opts:) Stopping client oleg303-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg303-server' oleg303-server: oleg303-server.virtnet: executing load_modules_local oleg303-server: kmemleak disabled oleg303-server: Loading modules from /home/green/git/lustre-release/lustre oleg303-server: detected 4 online CPUs by sysfs oleg303-server: MODOPTS_LIBCFS= oleg303-server: Force libcfs to create 2 CPU partitions oleg303-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre2 PASS 109 (44s) == sanityn test 110: do not grant another lock on resend ========================================================== 20:56:35 (1788828995) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272414920707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:56:36.000000000 -0400 Modify: 2026-09-07 20:56:36.000000000 -0400 Change: 2026-09-07 20:56:36.000000000 -0400 Birth: 2026-09-07 20:56:37.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg303-client.virtnet oleg303-client.virtnet PASS 110 (170s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 20:59:25 (1788829165) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (3s) == sanityn test 112: update max-inherit in default LMV === 20:59:28 (1788829168) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (2s) == sanityn test 113: check servers of specified fs ======= 20:59:30 (1788829170) lustre-OST0000-osc-ffff9131d83fc000 active. lustre-OST0001-osc-ffff9131d83fc000 active. lustre-OST0000-osc-ffff9131d83fc000 active. lustre-OST0001-osc-ffff9131d83fc000 active. MGC192.168.203.103@tcp active. PASS 113 (4s) == sanityn test 114: implicit default LMV inherit ======== 20:59:34 (1788829174) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (2s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 20:59:37 (1788829177) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (2s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 20:59:39 (1788829179) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (2s) == sanityn test 121: trunc append race =================== 20:59:41 (1788829181) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0137336 s, 1.5 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0626437 s, 327 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (7s) == sanityn test 122: directory size is consistent across mounts ========================================================== 20:59:48 (1788829188) PASS 122 (6s) == sanityn test 200: service remains healthy while able to process request ========================================================== 20:59:54 (1788829194) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=12 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0616 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0646 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0944 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1051 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1181 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1818 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1624 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1758 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.185 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1276 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0847 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1569 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1771 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2056 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1479 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1397 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1353 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.22 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.179 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2361 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2671 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2836 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3448 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3588 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg303-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9131d83fc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9131d83fc000.ost_server_uuid in FULL state after 0 sec PASS 200 (74s) cleanup: ====================================================== == sanityn test complete, duration 9386 sec ============== 21:01:09 (1788829269) === sanityn: start cleanup 21:01:10 (1788829270) === Stopping clients: oleg303-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg303-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 21:03:51 (1788829431) ===