-----============= acceptance-small: sanityn ============----- Fri Sep 4 11:11:21 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg630-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 11:11:43 (1788534703) === Starting client oleg630-client.virtnet: -o user_xattr,flock 192.168.206.130@tcp:/lustre /mnt/lustre2 Mount client oleg630-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.206.130@tcp:/lustre /mnt/lustre2 Started clients oleg630-client.virtnet: 192.168.206.130@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg630-client.virtnet: executing check_config_client /mnt/lustre oleg630-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg630-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9569856e6800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9569926b4800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9569856e6800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9569926b4800.idle_timeout=debug disable quota as required oleg630-server: oleg630-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 11:12:18 (1788534738) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685718 s, 15.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7612] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 11:12:21 (1788534741) PASS 0 (9s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 11:12:30 (1788534750) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 11:12:37 (1788534757) -rw-r--r-- 1 root root 0 Sep 4 11:12 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 11:12:45 (1788534765) -rw-r--r-- 1 root root 0 Sep 4 11:12 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (8s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 11:12:53 (1788534773) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (8s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 11:13:01 (1788534781) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (8s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 11:13:10 (1788534790) total 1 -rwxrwxrwx 1 root root 0 Sep 4 11:12 f2a -rwxrwxrwx 1 root root 0 Sep 4 11:12 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 4 11:12 f2a -rwxrwxrwx 1 root root 0 Sep 4 11:12 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.010729 s, 47.7 kB/s PASS 2e (10s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 11:13:19 (1788534799) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 11:13:23 (1788534803) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.456729 s, 4.6 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 11:13:30 (1788534810) PASS 3 (6s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 11:13:37 (1788534817) PASS 4 (7s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 11:13:43 (1788534823) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 11:13:50 (1788534830) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (6s) == sanityn test 7: remove of open directory on other node ======================================================================= 11:13:56 (1788534836) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (7s) == sanityn test 8: remove of open special file on other node ==================================================================== 11:14:03 (1788534843) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (6s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 11:14:09 (1788534849) PASS 9a (9s) == sanityn test 9b: append to striped sparse file ======== 11:14:18 (1788534858) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339731 s, 30.9 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0319897 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 11:14:25 (1788534865) 1+0 records in 1+0 records out 1 byte copied, 0.0084784 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0312664 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.013876 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0153343 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.011564 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0104883 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0256929 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0105478 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0185573 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0121882 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0170831 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0137032 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 11:14:33 (1788534873) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0512681 s, 59.9 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0655543 s, 62.5 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000373902 s, 8.2 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 11:14:40 (1788534880) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 11:14:47 (1788534887) warning: '-runtime' deprecated, use '-t runtime' instead - link 296 (time 1788534899.50 total 10.01 last 29.56) - link 607 (time 1788534909.51 total 20.02 last 31.07) - link 898 (time 1788534919.52 total 30.03 last 29.09) - link 1197 (time 1788534929.52 total 40.03 last 29.88) - link 1501 (time 1788534939.54 total 50.05 last 30.35) - link 1799 (time 1788534949.54 total 60.05 last 29.79) - link 2095 (time 1788534959.57 total 70.08 last 29.52) - link 2458 (time 1788534969.57 total 80.08 last 36.28) - link 2773 (time 1788534979.57 total 90.08 last 31.49) total: 3051 link in 100.02 seconds: 30.50 ops/second using seed 914423723 running for 100 seconds - stat 4638 (time 1788534901 ; total 11 ; last 11) - stat 10000 (time 1788534911 ; total 21 ; last 10) - stat 15197 (time 1788534922 ; total 32 ; last 11) - stat 20000 (time 1788534931 ; total 41 ; last 9) - stat 24840 (time 1788534942 ; total 52 ; last 11) - stat 27522 (time 1788534953 ; total 63 ; last 11) - stat 30000 (time 1788534963 ; total 73 ; last 10) - stat 32567 (time 1788534974 ; total 84 ; last 11) - stat 35182 (time 1788534985 ; total 95 ; last 11) total: 36388 stats in 100 seconds: 363.880005 stats/second - unlinked 0 (time 1788534940 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3051) error: No such file or directory total: 3050 unlinks in 84 seconds: 36.309525 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21414) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21416) - No such process PASS 12 (144s) == sanityn test 13: test directory page revocation ======= 11:17:11 (1788535031) f13 f13 PASS 13 (10s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 11:17:21 (1788535041) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22602 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 11:17:29 (1788535049) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23181 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 11:17:37 (1788535057) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23760 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 11:17:44 (1788535064) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24342 Terminated $DIR1/$tdir/sleep 60 PASS 14c (8s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 11:17:52 (1788535072) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24924 Terminated $DIR1/$tdir/sleep 60 PASS 14d (10s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 11:18:02 (1788535082) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (56s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 11:18:58 (1788535138) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.575881 s, 14.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.455232 s, 18.4 MB/s Chance of close/open is 1 in 50 Seed set to 5143 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000005[0] 1788535144.861443 mapwrite 0x5a6d60 thru 0x5a717a (0x41b bytes) truncating to largest ever: 0xc8d6c 000008[1] 1788535145.073568 trunc from 0x5a717b to 0x0c8d6b (0x4de40f bytes) truncating to largest ever: 0x595c51 000011[1] 1788535145.279962 trunc from 0x0c8d6c to 0x595c50 (0x4ccee5 bytes) 000013[0] 1788535145.383560 mapwrite 0x295100 thru 0x29958b (0x448c bytes) 000014[0] 1788535145.543361 mapread 0x51dd1e thru 0x52a8f2 (0xcbd5 bytes) 000015[0] 1788535145.626604 read 0x3c5cb7 thru 0x3c8bd9 (0x2f23 bytes) 000019[1] 1788535145.725038 read 0x316e48 thru 0x32193d (0xaaf6 bytes) 000020[0] 1788535145.768051 mapread 0x14f4b8 thru 0x1552a4 (0x5ded bytes) 000022[0] 1788535145.816915 write 0x55bea0 thru 0x5627f9 (0x695a bytes) 000023[0] 1788535145.877107 trunc from 0x595c51 to 0x4677fb (0x12e455 bytes) truncating to largest ever: 0x5bb3e3 000024[0] 1788535145.961807 trunc from 0x4677fc to 0x5bb3e2 (0x153be7 bytes) 000025[1] 1788535146.011628 mapread 0x016197 thru 0x017209 (0x1073 bytes) 000026[1] 1788535146.066522 punch from 0x332709 to 0x33bd1f (0x9617 bytes) truncating to largest ever: 0x723f3d 000027[1] 1788535146.121292 trunc from 0x5bb3e3 to 0x723f3c (0x168b5a bytes) 000029[0] 1788535146.206641 mapread 0x52b7f9 thru 0x52d6fa (0x1f02 bytes) 000030[0] 1788535146.274123 punch from 0x2c34cc to 0x2c9dfe (0x6933 bytes) 000033[0] 1788535146.360275 punch from 0x0ea6ac to 0x0f0171 (0x5ac6 bytes) 000034[1] 1788535146.401004 read 0x1f009f thru 0x1fd5bc (0xd51e bytes) 000035[1] 1788535146.433092 mapread 0x2fdef9 thru 0x30c768 (0xe870 bytes) 000036[1] 1788535146.496929 read 0x295a51 thru 0x29f90b (0x9ebb bytes) 000037[0] 1788535146.506896 read 0x2ccf4c thru 0x2cdbe1 (0xc96 bytes) 000039[0] 1788535146.527133 read 0x05bbe3 thru 0x0685ab (0xc9c9 bytes) 000040[0] 1788535146.544583 mapread 0x3090fa thru 0x30ce92 (0x3d99 bytes) 000041[1] 1788535146.578773 trunc from 0x723f3d to 0x40a4d1 (0x319a6b bytes) 000042[1] 1788535146.636653 write 0x2e6774 thru 0x2e98bc (0x3149 bytes) 000044[1] 1788535146.676744 mapwrite 0x692ff0 thru 0x695ffa (0x300b bytes) 000045[1] 1788535146.782183 mapwrite 0x9ab082 thru 0x9b92a5 (0xe224 bytes) 000046[1] 1788535146.978028 punch from 0x91dd2b to 0x92c952 (0xec28 bytes) 000048[1] 1788535146.983912 read 0x8b3b14 thru 0x8bcb2f (0x901c bytes) 000049[0] 1788535146.995596 read 0x4c1076 thru 0x4c6b30 (0x5abb bytes) 000050[0] 1788535147.052145 trunc from 0x9b92a6 to 0x05d3b8 (0x95beed bytes) 000052[0] 1788535147.126181 punch from 0x0166f2 to 0x01c5c1 (0x5ed0 bytes) 000054[1] 1788535147.164295 write 0x861799 thru 0x86186e (0xd6 bytes) 000055[1] 1788535147.204800 write 0x201180 thru 0x202265 (0x10e6 bytes) 000056[1] 1788535147.237745 read 0x711c1b thru 0x714c60 (0x3046 bytes) 000057[0] 1788535147.260948 mapwrite 0x7a7643 thru 0x7b2556 (0xaf14 bytes) 000058[0] 1788535147.735201 trunc from 0x86186f to 0x60b2c0 (0x2565ae bytes) 000059[0] 1788535147.786083 mapwrite 0x4e2a3b thru 0x4ee7ff (0xbdc5 bytes) 000061[1] 1788535147.901749 punch from 0x095ee2 to 0x09aa8e (0x4bad bytes) 000064[1] 1788535147.970754 read 0x1d8b90 thru 0x1e42d9 (0xb74a bytes) 000066[1] 1788535148.030290 punch from 0x250801 to 0x25c5ca (0xbdca bytes) 000068[0] 1788535148.105165 trunc from 0x60b2c1 to 0x133844 (0x4d7a7c bytes) 000069[0] 1788535148.169668 mapread 0x0952fa thru 0x09d98d (0x8694 bytes) 000070[1] 1788535148.239322 mapread 0x096726 thru 0x0a14fe (0xadd9 bytes) 000071[0] 1788535148.318280 mapwrite 0x9d4cd3 thru 0x9deb25 (0x9e53 bytes) truncating to largest ever: 0x8f459e 000073[0] 1788535148.569414 trunc from 0x9deb26 to 0x8f459d (0xea588 bytes) 000075[0] 1788535148.661653 mapwrite 0x4cb86c thru 0x4cd301 (0x1a96 bytes) 000076[0] 1788535148.800572 mapread 0x3b06df thru 0x3b28a3 (0x21c5 bytes) 000077[1] 1788535148.829320 write 0x2db23e thru 0x2e1cb7 (0x6a7a bytes) 000079[1] 1788535148.924652 trunc from 0x8f459e to 0x8547cd (0x9fdd0 bytes) 000081[1] 1788535149.026497 trunc from 0x8547ce to 0x72485a (0x12ff73 bytes) 000082[1] 1788535149.048207 trunc from 0x72485b to 0x3f8e57 (0x32ba03 bytes) 000084[0] 1788535149.096658 mapwrite 0x72b1eb thru 0x738ad5 (0xd8eb bytes) 000085[1] 1788535149.283601 read 0x528538 thru 0x534319 (0xbde2 bytes) 000086[1] 1788535149.349589 punch from 0x25f896 to 0x2637cb (0x3f36 bytes) 000088[1] 1788535149.398588 mapwrite 0x23bc09 thru 0x23fab3 (0x3eab bytes) 000090[1] 1788535149.576247 read 0x271cd6 thru 0x27f712 (0xda3d bytes) 000091[1] 1788535149.589124 punch from 0x699201 to 0x6a5045 (0xbe45 bytes) 000093[1] 1788535149.633152 mapread 0x31aa5e thru 0x31e9a1 (0x3f44 bytes) 000095[0] 1788535149.645731 trunc from 0x738ad6 to 0x12b48a (0x60d64b bytes) 000096[1] 1788535149.729927 punch from 0x095353 to 0x09b635 (0x62e3 bytes) 000097[0] 1788535149.763515 write 0x739a34 thru 0x746aed (0xd0ba bytes) 000101[0] 1788535149.838675 trunc from 0x746aee to 0x173846 (0x5d32a7 bytes) 000103[1] 1788535149.926012 mapread 0x14a8c5 thru 0x152497 (0x7bd3 bytes) 000105[0] 1788535150.027148 write 0x822810 thru 0x822f6c (0x75d bytes) 000107[1] 1788535150.084784 punch from 0x1646fc to 0x16ac1b (0x6520 bytes) 000111[0] 1788535150.173072 punch from 0x4e1e90 to 0x4ee573 (0xc6e4 bytes) 000112[0] 1788535150.195678 read 0x5df81d thru 0x5ec526 (0xcd0a bytes) 000115[0] 1788535150.343358 punch from 0x63747a to 0x6406a8 (0x922f bytes) 000116[1] 1788535150.364299 trunc from 0x822f6d to 0x5f6c86 (0x22c2e6 bytes) 000118[1] 1788535150.424936 write 0x576a50 thru 0x584850 (0xde01 bytes) 000119[1] 1788535150.454350 read 0x3287aa thru 0x331d16 (0x956d bytes) 000121[1] 1788535150.487043 mapread 0x2267f9 thru 0x22752c (0xd34 bytes) 000122[0] 1788535150.521648 mapread 0x18e83a thru 0x198eba (0xa681 bytes) 000123[0] 1788535150.580850 write 0x3a1890 thru 0x3ab592 (0x9d03 bytes) 000124[0] 1788535150.645414 mapwrite 0x4c5df7 thru 0x4c9f05 (0x410f bytes) 000125[0] 1788535150.968930 mapwrite 0x13c15e thru 0x144380 (0x8223 bytes) 000126[0] 1788535151.046487 punch from 0x10724f to 0x10f99d (0x874f bytes) 000127[0] 1788535151.074708 punch from 0x581b01 to 0x5891d8 (0x76d8 bytes) 000129[0] 1788535151.080673 mapread 0x381762 thru 0x38851d (0x6dbc bytes) 000131[1] 1788535151.159179 trunc from 0x5f6c87 to 0x622183 (0x2b4fd bytes) 000132[0] 1788535151.208156 write 0x02b817 thru 0x02f5f7 (0x3de1 bytes) 000133[0] 1788535151.270582 read 0x02dbb5 thru 0x037f0d (0xa359 bytes) 000134[1] 1788535151.302065 read 0x206080 thru 0x2137b1 (0xd732 bytes) 000136[1] 1788535151.362483 trunc from 0x622184 to 0x13f008 (0x4e317b bytes) 000139[0] 1788535151.488880 write 0x31f279 thru 0x32d420 (0xe1a8 bytes) 000142[0] 1788535151.638750 write 0x62d9e5 thru 0x634574 (0x6b90 bytes) 000146[0] 1788535151.751687 read 0x11d27a thru 0x1206e0 (0x3467 bytes) 000148[0] 1788535151.818396 mapwrite 0x776b75 thru 0x77b356 (0x47e2 bytes) 000149[1] 1788535152.020471 read 0x5c4ee3 thru 0x5c5207 (0x325 bytes) 000150[0] 1788535152.049071 mapwrite 0x94c2f4 thru 0x950f66 (0x4c73 bytes) 000151[0] 1788535152.195030 mapwrite 0x97c25a thru 0x982111 (0x5eb8 bytes) 000152[1] 1788535152.320173 write 0x20f861 thru 0x21328f (0x3a2f bytes) 000153[0] 1788535152.432297 write 0x12fd81 thru 0x136f04 (0x7184 bytes) 000154[0] 1788535152.544267 read 0x6ed345 thru 0x6ef98c (0x2648 bytes) 000155[0] 1788535152.566995 write 0x92bbd5 thru 0x92d75f (0x1b8b bytes) 000159[0] 1788535152.796101 trunc from 0x982112 to 0x827384 (0x15ad8d bytes) 000160[1] 1788535152.844472 mapread 0x708024 thru 0x7131f8 (0xb1d5 bytes) 000161[0] 1788535153.055721 punch from 0x122f22 to 0x12e430 (0xb50f bytes) 000162[0] 1788535153.102451 mapread 0x77bbc2 thru 0x77c5ea (0xa29 bytes) 000163[1] 1788535153.143579 mapread 0x5db09f thru 0x5e465d (0x95bf bytes) 000165[1] 1788535153.191604 punch from 0x152cf6 to 0x15c078 (0x9383 bytes) 000166[1] 1788535153.244458 mapread 0x0df7fb thru 0x0e18ca (0x20d0 bytes) 000167[0] 1788535153.302099 trunc from 0x827385 to 0x818006 (0xf37e bytes) 000169[1] 1788535153.381744 trunc from 0x818007 to 0x2ea9c5 (0x52d641 bytes) 000171[0] 1788535153.533714 punch from 0x023251 to 0x026c39 (0x39e9 bytes) 000172[0] 1788535153.583683 punch from 0x226227 to 0x22ab41 (0x491b bytes) 000173[0] 1788535153.626508 mapwrite 0x410b48 thru 0x411698 (0xb51 bytes) 000175[1] 1788535153.788541 read 0x073c18 thru 0x07dd98 (0xa181 bytes) truncating to largest ever: 0x962bbc 000176[0] 1788535153.830771 trunc from 0x411699 to 0x962bbb (0x551523 bytes) 000177[0] 1788535153.906121 mapwrite 0x246035 thru 0x251d41 (0xbd0d bytes) 000178[0] 1788535154.002791 write 0x08e018 thru 0x09a222 (0xc20b bytes) 000179[1] 1788535154.040239 mapwrite 0x9403bd thru 0x9459c9 (0x560d bytes) 000181[0] 1788535154.275621 write 0x16a70f thru 0x16b01e (0x910 bytes) 000182[1] 1788535154.312606 read 0x0a3e18 thru 0x0af064 (0xb24d bytes) 000184[1] 1788535154.420574 read 0x0f921b thru 0x0fa32f (0x1115 bytes) 000185[0] 1788535154.458278 mapwrite 0x5f96eb thru 0x606e71 (0xd787 bytes) 000187[0] 1788535154.627807 punch from 0x2e3aef to 0x2e799c (0x3eae bytes) 000188[1] 1788535154.656320 trunc from 0x962bbc to 0x38cd27 (0x5d5e94 bytes) 000189[0] 1788535154.735765 mapread 0x175d47 thru 0x177000 (0x12ba bytes) 000190[1] 1788535154.841818 write 0x8ba886 thru 0x8ca52d (0xfca8 bytes) truncating to largest ever: 0x9d5249 000192[1] 1788535154.900361 trunc from 0x8ca52e to 0x9d5248 (0x10ad1b bytes) 000193[1] 1788535154.929501 trunc from 0x9d5249 to 0x2477eb (0x78da5d bytes) 000197[0] 1788535155.018687 mapread 0x231ec9 thru 0x235410 (0x3548 bytes) 000203[1] 1788535155.089230 mapwrite 0x5ab317 thru 0x5b7a4d (0xc737 bytes) 000204[0] 1788535155.213663 trunc from 0x5b7a4e to 0x382ab5 (0x234f98 bytes) 000205[1] 1788535155.296309 read 0x22dfe8 thru 0x23167c (0x3695 bytes) 000208[1] 1788535155.514010 write 0x161778 thru 0x16b0f7 (0x9980 bytes) 000211[1] 1788535155.670151 mapread 0x0574bf thru 0x05cf36 (0x5a78 bytes) 000212[0] 1788535155.749320 read 0x102819 thru 0x103ded (0x15d5 bytes) 000213[0] 1788535155.872661 trunc from 0x382ab6 to 0x9c054b (0x63da96 bytes) 000214[1] 1788535155.934186 trunc from 0x9c054c to 0x54292d (0x47dc1e bytes) 000215[0] 1788535155.994557 mapwrite 0x48afd8 thru 0x492b6d (0x7b96 bytes) 000216[1] 1788535156.119735 trunc from 0x54292e to 0x57440b (0x31ade bytes) 000219[0] 1788535156.192343 trunc from 0x57440c to 0x0fd0ce (0x47733d bytes) 000220[0] 1788535156.265703 read 0x043bee thru 0x04633c (0x274f bytes) 000221[1] 1788535156.326578 mapread 0x0e1e14 thru 0x0e25a5 (0x792 bytes) 000222[1] 1788535156.364660 mapwrite 0x3d7c69 thru 0x3e039d (0x8735 bytes) 000224[1] 1788535156.506585 mapread 0x273713 thru 0x273c4e (0x53c bytes) 000225[1] 1788535156.553438 trunc from 0x3e039e to 0x54ff55 (0x16fbb8 bytes) 000229[1] 1788535156.625236 mapread 0x31c1cd thru 0x31c9ff (0x833 bytes) 000230[0] 1788535156.661088 mapwrite 0x3be5ff thru 0x3c2225 (0x3c27 bytes) 000231[0] 1788535156.982831 write 0x6f9a27 thru 0x702c32 (0x920c bytes) 000232[1] 1788535157.001483 read 0x083602 thru 0x08e6a6 (0xb0a5 bytes) 000233[1] 1788535157.067793 punch from 0x1e4679 to 0x1e841d (0x3da5 bytes) 000234[0] 1788535157.137755 trunc from 0x702c33 to 0x852c32 (0x150000 bytes) 000235[0] 1788535157.194629 write 0x9b2752 thru 0x9bab5d (0x840c bytes) 000236[1] 1788535157.251662 read 0x4c3884 thru 0x4cd692 (0x9e0f bytes) 000237[1] 1788535157.331137 mapread 0x4fd994 thru 0x4fdc0b (0x278 bytes) 000239[1] 1788535157.418177 trunc from 0x9bab5e to 0x238c67 (0x781ef6 bytes) 000240[1] 1788535157.546958 mapwrite 0x82c979 thru 0x83325e (0x68e6 bytes) 000243[1] 1788535157.822170 mapwrite 0x33e60e thru 0x34a0b1 (0xbaa4 bytes) 000245[0] 1788535158.030013 mapwrite 0x9a33bd thru 0x9b01a4 (0xcde8 bytes) 000246[1] 1788535158.344166 read 0x8b009e thru 0x8bebdb (0xeb3e bytes) 000249[1] 1788535158.427118 write 0x8cbc69 thru 0x8d5bb6 (0x9f4e bytes) 000250[1] 1788535158.470812 punch from 0x8a5d13 to 0x8a6816 (0xb04 bytes) 000252[0] 1788535158.523592 mapwrite 0x05e1c2 thru 0x061403 (0x3242 bytes) 000253[1] 1788535158.638323 mapread 0x4eb6aa thru 0x4f7d10 (0xc667 bytes) 000254[1] 1788535158.717146 mapread 0x67d580 thru 0x684a9b (0x751c bytes) 000255[0] 1788535158.748483 punch from 0x669be7 to 0x6766bb (0xcad5 bytes) 000256[1] 1788535158.785192 write 0x8f17ff thru 0x8f7e82 (0x6684 bytes) 000257[1] 1788535158.818554 read 0x7ebf63 thru 0x7f96d6 (0xd774 bytes) 000259[1] 1788535158.839650 write 0x36224c thru 0x36c564 (0xa319 bytes) 000261[1] 1788535158.865772 read 0x1c2e9d thru 0x1cb8e1 (0x8a45 bytes) 000262[0] 1788535158.876766 mapwrite 0x7658da thru 0x771b1a (0xc241 bytes) 000265[0] 1788535159.064351 mapwrite 0x8b2517 thru 0x8bd8fe (0xb3e8 bytes) 000266[0] 1788535159.259505 trunc from 0x9b01a5 to 0x518923 (0x497881 bytes) 000268[1] 1788535159.319429 punch from 0x0d993a to 0x0e5571 (0xbc38 bytes) 000270[1] 1788535159.394841 mapread 0x101235 thru 0x10befa (0xacc6 bytes) 000271[1] 1788535159.448009 punch from 0x3129a3 to 0x321c76 (0xf2d4 bytes) 000272[0] 1788535159.472455 mapread 0x179b17 thru 0x186ac7 (0xcfb1 bytes) 000273[1] 1788535159.626476 mapread 0x2ccf8a thru 0x2d1065 (0x40dc bytes) 000274[1] 1788535159.680583 punch from 0x100c3a to 0x104975 (0x3d3c bytes) 000276[1] 1788535159.754114 mapread 0x2aeb7a thru 0x2bc33b (0xd7c2 bytes) 000279[0] 1788535159.867011 mapread 0x01ff00 thru 0x02e7d9 (0xe8da bytes) 000282[0] 1788535160.003472 trunc from 0x518924 to 0x74b3dc (0x232ab9 bytes) 000283[0] 1788535160.080501 mapread 0x18e586 thru 0x199185 (0xac00 bytes) 000285[0] 1788535160.285528 trunc from 0x74b3dd to 0x1bc952 (0x58ea8a bytes) 000286[1] 1788535160.385363 write 0x720ea0 thru 0x728d7e (0x7edf bytes) 000287[1] 1788535160.462169 mapwrite 0x0899d1 thru 0x08e001 (0x4631 bytes) 000288[0] 1788535160.593676 write 0x138713 thru 0x147481 (0xed6f bytes) 000290[1] 1788535160.652997 read 0x0b1665 thru 0x0b1a3b (0x3d7 bytes) 000291[0] 1788535160.718673 read 0x1aba0e thru 0x1b6cbf (0xb2b2 bytes) 000295[1] 1788535160.818971 trunc from 0x728d7f to 0x5799bb (0x1af3c3 bytes) 000297[1] 1788535160.860692 mapwrite 0x62b9d8 thru 0x62d707 (0x1d30 bytes) 000298[0] 1788535160.977594 mapread 0x53a58f thru 0x53b026 (0xa98 bytes) 000299[1] 1788535161.020476 mapread 0x572647 thru 0x57a7e3 (0x819d bytes) 000300[1] 1788535161.059398 write 0x690545 thru 0x69a95a (0xa416 bytes) 000301[1] 1788535161.103759 write 0x92746b thru 0x92ef56 (0x7aec bytes) 000304[0] 1788535161.155737 punch from 0x596751 to 0x59ac1a (0x44ca bytes) 000305[1] 1788535161.195431 trunc from 0x92ef57 to 0x184daf (0x7aa1a7 bytes) 000306[0] 1788535161.235009 read 0x17ad6b thru 0x184daf (0xa045 bytes) 000308[0] 1788535161.260847 trunc from 0x184db0 to 0x0dc444 (0xa896b bytes) 000309[0] 1788535161.319411 read 0x0839d3 thru 0x0904e7 (0xcb15 bytes) 000310[1] 1788535161.364610 mapwrite 0x85e6b3 thru 0x86ab90 (0xc4de bytes) 000311[0] 1788535161.529423 mapread 0x6d0446 thru 0x6df566 (0xf121 bytes) 000313[0] 1788535161.648581 mapread 0x0cae5c thru 0x0cec21 (0x3dc6 bytes) truncating to largest ever: 0x9eea1f 000316[1] 1788535161.762844 trunc from 0x86ab91 to 0x9eea1e (0x183e8e bytes) 000317[1] 1788535161.850472 read 0x94539c thru 0x95419a (0xedff bytes) 000318[0] 1788535161.885473 read 0x5facb1 thru 0x5fb2a0 (0x5f0 bytes) 000319[0] 1788535161.947771 mapread 0x2f9510 thru 0x301330 (0x7e21 bytes) 000320[1] 1788535162.037045 mapwrite 0x8f43b3 thru 0x8fde37 (0x9a85 bytes) 000321[0] 1788535162.194689 mapread 0x090c8d thru 0x091155 (0x4c9 bytes) 000322[0] 1788535162.253723 mapread 0x55f49c thru 0x56d803 (0xe368 bytes) 000323[1] 1788535162.302250 mapwrite 0x0d9382 thru 0x0dce73 (0x3af2 bytes) 000324[0] 1788535162.402608 write 0x8e6e3e thru 0x8f17ed (0xa9b0 bytes) 000325[0] 1788535162.426694 trunc from 0x9eea1f to 0x943311 (0xab70d bytes) 000326[1] 1788535162.484527 punch from 0x4e8b9c to 0x4f7959 (0xedbe bytes) 000328[0] 1788535162.546642 mapwrite 0x1830a1 thru 0x18e2b6 (0xb216 bytes) 000331[0] 1788535162.737621 trunc from 0x943312 to 0x3446d7 (0x5fec3a bytes) 000333[1] 1788535162.783433 write 0x9ffb5c thru 0x9fffff (0x4a4 bytes) 000338[1] 1788535162.950748 mapwrite 0x55d9f9 thru 0x55fab8 (0x20c0 bytes) 000341[0] 1788535163.059935 read 0x02690b thru 0x033acf (0xd1c5 bytes) 000343[1] 1788535163.114681 mapread 0x3f7c7c thru 0x4074ac (0xf831 bytes) 000347[0] 1788535163.197726 mapwrite 0x489ab5 thru 0x48ea28 (0x4f74 bytes) 000348[1] 1788535163.354073 trunc from 0xa00000 to 0x768395 (0x297c6a bytes) 000350[0] 1788535163.445619 write 0x8d4493 thru 0x8e25be (0xe12c bytes) 000352[1] 1788535163.505832 read 0x84232c thru 0x85229b (0xff70 bytes) 000353[0] 1788535163.537132 read 0x482831 thru 0x485a22 (0x31f2 bytes) 000355[1] 1788535163.568474 trunc from 0x8e25bf to 0x57659e (0x36c020 bytes) 000356[0] 1788535163.644271 write 0x7e1801 thru 0x7ed4ee (0xbcee bytes) 000359[1] 1788535163.736547 mapread 0x45f5c4 thru 0x46f002 (0xfa3f bytes) 000360[1] 1788535163.801450 read 0x1d5273 thru 0x1dbe5b (0x6be9 bytes) 000362[1] 1788535163.821493 mapwrite 0x8f3616 thru 0x8fe1ea (0xabd5 bytes) 000364[0] 1788535163.957328 mapread 0x0f76bf thru 0x0f8b1d (0x145f bytes) 000365[1] 1788535164.008340 mapread 0x6c6d36 thru 0x6ce7d2 (0x7a9d bytes) 000370[0] 1788535164.070313 punch from 0x623cda to 0x62bf3b (0x8262 bytes) 000371[0] 1788535164.085729 mapread 0x5d7eea thru 0x5e2ab9 (0xabd0 bytes) 000372[1] 1788535164.117740 mapwrite 0x64a5d2 thru 0x64d3f8 (0x2e27 bytes) 000373[1] 1788535164.191293 punch from 0x2dfacc to 0x2ed232 (0xd767 bytes) 000374[1] 1788535164.231032 write 0x8a3a08 thru 0x8a6981 (0x2f7a bytes) 000375[0] 1788535164.263349 trunc from 0x8fe1eb to 0x402deb (0x4fb3ff bytes) 000376[1] 1788535164.389764 write 0x7dc7c2 thru 0x7dcfe0 (0x81f bytes) 000380[0] 1788535164.539412 mapwrite 0x092b85 thru 0x09b2e8 (0x8764 bytes) 000382[1] 1788535164.794383 write 0x55bc10 thru 0x56b1c7 (0xf5b8 bytes) 000384[0] 1788535164.882849 read 0x18ef18 thru 0x18ff0a (0xff3 bytes) 000386[0] 1788535164.983431 write 0x21940f thru 0x227774 (0xe366 bytes) 000387[1] 1788535165.045804 read 0x328380 thru 0x32842c (0xad bytes) 000389[0] 1788535165.163450 mapwrite 0x82ea4e thru 0x8395a5 (0xab58 bytes) 000391[1] 1788535165.429481 mapwrite 0x68c63a thru 0x69b668 (0xf02f bytes) 000392[1] 1788535165.570356 trunc from 0x8395a6 to 0x3ee889 (0x44ad1c bytes) 000393[1] 1788535165.642748 punch from 0x053137 to 0x0551fa (0x20c4 bytes) 000394[0] 1788535165.680723 mapread 0x2cec81 thru 0x2d694c (0x7ccc bytes) 000395[0] 1788535165.738840 mapread 0x2f99bf thru 0x2fd98c (0x3fce bytes) 000396[0] 1788535165.777042 trunc from 0x3ee88a to 0x27fe53 (0x16ea36 bytes) 000397[1] 1788535165.848646 punch from 0x1e1431 to 0x1eda12 (0xc5e2 bytes) 000398[1] 1788535165.883184 punch from 0x1f2c64 to 0x1fc93a (0x9cd7 bytes) 000399[1] 1788535165.920302 punch from 0x195525 to 0x19724e (0x1d2a bytes) 000401[0] 1788535165.992430 mapwrite 0x844fe6 thru 0x84827c (0x3297 bytes) 000402[1] 1788535166.110458 trunc from 0x84827d to 0x300f1a (0x547362 bytes) 000403[1] 1788535166.199673 mapread 0x166f1b thru 0x170adb (0x9bc1 bytes) 000406[0] 1788535166.268525 read 0x1052e9 thru 0x10deeb (0x8c03 bytes) 000407[0] 1788535166.372259 trunc from 0x300f1b to 0x6f32c3 (0x3f23a9 bytes) 000409[0] 1788535166.534266 mapread 0x3a3b0c thru 0x3b3024 (0xf519 bytes) 000411[0] 1788535166.670925 mapread 0x3c108e thru 0x3c74da (0x644d bytes) 000417[0] 1788535166.827730 punch from 0x2b3b35 to 0x2b5027 (0x14f3 bytes) 000419[1] 1788535166.882584 mapwrite 0x1c6e19 thru 0x1ca194 (0x337c bytes) 000420[0] 1788535167.066726 trunc from 0x6f32c4 to 0x6eb368 (0x7f5b bytes) 000423[0] 1788535167.190293 punch from 0x6cb3bd to 0x6d273c (0x7380 bytes) 000427[1] 1788535167.317489 mapread 0x158876 thru 0x160692 (0x7e1d bytes) 000428[1] 1788535167.395339 punch from 0x6b4433 to 0x6c43de (0xffac bytes) 000429[1] 1788535167.417189 mapread 0x21bd1d thru 0x21c608 (0x8ec bytes) 000431[0] 1788535167.433848 punch from 0x449d72 to 0x458ebf (0xf14e bytes) 000433[1] 1788535167.487609 mapread 0x426a31 thru 0x4282dd (0x18ad bytes) 000435[1] 1788535167.561130 mapread 0x4c6cf3 thru 0x4cd92b (0x6c39 bytes) 000437[1] 1788535167.589848 read 0x12a92d thru 0x12c037 (0x170b bytes) 000438[0] 1788535167.596583 mapwrite 0x7c98aa thru 0x7cbe3e (0x2595 bytes) 000439[0] 1788535167.771125 mapread 0x6fcef0 thru 0x70318b (0x629c bytes) 000441[0] 1788535167.811693 mapread 0x19fe81 thru 0x1a6c1d (0x6d9d bytes) 000442[1] 1788535167.852661 trunc from 0x7cbe3f to 0x7165c2 (0xb587c bytes) 000444[1] 1788535167.910682 read 0x4fccd1 thru 0x507279 (0xa5a9 bytes) 000446[0] 1788535167.981723 write 0x8a0139 thru 0x8a39bf (0x3887 bytes) 000448[0] 1788535168.045899 mapwrite 0x3e8bde thru 0x3f6714 (0xdb37 bytes) 000450[0] 1788535168.256318 read 0x3f3109 thru 0x4028c0 (0xf7b8 bytes) 000452[0] 1788535168.315176 punch from 0x65bee6 to 0x66224a (0x6365 bytes) 000453[0] 1788535168.333084 mapread 0x0502c4 thru 0x05df7a (0xdcb7 bytes) 000454[1] 1788535168.383437 mapwrite 0x6869c7 thru 0x68827f (0x18b9 bytes) 000455[0] 1788535168.508382 read 0x2433a8 thru 0x24b8a5 (0x84fe bytes) 000457[1] 1788535168.569399 punch from 0x1c0fc1 to 0x1d0c2d (0xfc6d bytes) 000458[0] 1788535168.593273 punch from 0x489cb9 to 0x491ce1 (0x8029 bytes) 000459[0] 1788535168.662777 read 0x5e6e32 thru 0x5ebe12 (0x4fe1 bytes) 000463[0] 1788535168.792366 mapwrite 0x9e1972 thru 0x9e4178 (0x2807 bytes) 000464[0] 1788535168.960167 punch from 0x205e7f to 0x213373 (0xd4f5 bytes) 000467[1] 1788535168.976006 trunc from 0x9e4179 to 0x9a9950 (0x3a828 bytes) 000468[0] 1788535169.056489 read 0x659cff thru 0x65a450 (0x752 bytes) 000469[1] 1788535169.101563 punch from 0x98ad7a to 0x98da1c (0x2ca3 bytes) 000470[1] 1788535169.107606 mapread 0x1c443c thru 0x1cf632 (0xb1f7 bytes) 000472[0] 1788535169.175331 trunc from 0x9a9951 to 0x8e2ef9 (0xc6a57 bytes) 000473[1] 1788535169.261579 mapread 0x8b5a16 thru 0x8bbd6a (0x6355 bytes) 000476[0] 1788535169.341697 write 0x8d46a8 thru 0x8d8136 (0x3a8f bytes) 000477[1] 1788535169.372645 write 0x96ee63 thru 0x978134 (0x92d2 bytes) 000478[0] 1788535169.418961 write 0x57274c thru 0x57f1ff (0xcab4 bytes) 000479[0] 1788535169.449680 write 0x1cb8a6 thru 0x1d8d18 (0xd473 bytes) 000480[0] 1788535169.483084 trunc from 0x978135 to 0x0e1512 (0x896c22 bytes) 000482[1] 1788535169.505823 punch from 0x09ac8f to 0x0a77bf (0xcb31 bytes) 000483[0] 1788535169.540126 mapwrite 0x6c87c8 thru 0x6cb719 (0x2f52 bytes) 000485[1] 1788535169.633479 mapwrite 0x040c7a thru 0x04aa62 (0x9de9 bytes) 000486[1] 1788535169.744170 write 0x8f20a3 thru 0x8fb079 (0x8fd7 bytes) truncating to largest ever: 0x9fb509 000487[1] 1788535169.760687 trunc from 0x8fb07a to 0x9fb508 (0x10048f bytes) 000490[1] 1788535169.841584 mapwrite 0x53829a thru 0x547af2 (0xf859 bytes) 000491[1] 1788535169.920787 trunc from 0x9fb509 to 0x00474d (0x9f6dbb bytes) 000493[1] 1788535169.970266 punch from 0x00048b to 0x00474d (0x42c3 bytes) 000496[1] 1788535169.996692 mapread 0x00409e thru 0x00474d (0x6b0 bytes) 000497[1] 1788535170.004620 write 0x472298 thru 0x47ec04 (0xc96d bytes) 000499[1] 1788535170.036821 mapwrite 0x61c0c9 thru 0x622799 (0x66d1 bytes) 000500[1] 1788535170.153307 mapwrite 0x21271f thru 0x21e6a5 (0xbf87 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 5170 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000007[1] 1788535171.320339 mapwrite 0x7de000 thru 0x7e2e74 (0x4e75 bytes) truncating to largest ever: 0x87fd0a 000009[1] 1788535171.548262 trunc from 0x7e2e75 to 0x87fd09 (0x9ce95 bytes) 000010[1] 1788535171.579491 mapread 0x07b000 thru 0x0884ab (0xd4ac bytes) 000011[0] 1788535171.613426 write_OD 0x3c6000 thru 0x3d1fff (0xc000 bytes) 000012[0] 1788535171.741105 punch from 0x7f08db to 0x8006d5 (0xfdfb bytes) 000015[0] 1788535171.828033 write_OD 0x45f000 thru 0x462fff (0x4000 bytes) 000016[1] 1788535171.891002 mapwrite 0x073000 thru 0x07dbcc (0xabcd bytes) 000017[1] 1788535172.025533 mapwrite 0x0d4000 thru 0x0dbd6d (0x7d6e bytes) 000018[1] 1788535172.204576 mapread 0x677000 thru 0x686a0c (0xfa0d bytes) 000019[1] 1788535172.275723 punch from 0x61a1fe to 0x627f07 (0xdd0a bytes) 000020[0] 1788535172.300400 mapwrite 0x185000 thru 0x18f330 (0xa331 bytes) 000022[0] 1788535172.405378 mapwrite 0x60e000 thru 0x61dc4d (0xfc4e bytes) 000023[1] 1788535172.508969 mapread 0x728000 thru 0x72f90e (0x790f bytes) 000024[0] 1788535172.544534 mapread 0x368000 thru 0x374f85 (0xcf86 bytes) 000026[1] 1788535172.574151 read_OD 0x739000 thru 0x744fff (0xc000 bytes) 000032[1] 1788535172.589582 punch from 0x0a9084 to 0x0b8140 (0xf0bd bytes) 000035[1] 1788535172.643560 mapread 0x5a5000 thru 0x5a6bd6 (0x1bd7 bytes) 000037[1] 1788535172.679686 read_OD 0x369000 thru 0x36cfff (0x4000 bytes) 000038[0] 1788535172.702726 mapread 0x23e000 thru 0x243851 (0x5852 bytes) 000039[1] 1788535172.725685 read_OD 0x44a000 thru 0x44cfff (0x3000 bytes) 000042[1] 1788535172.735504 trunc from 0x87fd0a to 0x0c670a (0x7b95ff bytes) 000043[0] 1788535172.800943 trunc from 0x0c670b to 0x3dd5bb (0x316eb1 bytes) 000044[1] 1788535172.844741 write_OD 0x167000 thru 0x170fff (0xa000 bytes) 000046[1] 1788535172.923065 write_OD 0x28f000 thru 0x294fff (0x6000 bytes) 000047[0] 1788535172.978645 punch from 0x2e34cc to 0x2e4479 (0xfae bytes) 000049[1] 1788535173.030267 read_OD 0x14b000 thru 0x150fff (0x6000 bytes) 000050[1] 1788535173.066535 mapread 0x308000 thru 0x312451 (0xa452 bytes) 000053[1] 1788535173.177735 trunc from 0x3dd5bc to 0x2f8d8e (0xe482d bytes) 000055[1] 1788535173.241370 mapread 0x03a000 thru 0x03b068 (0x1069 bytes) 000056[0] 1788535173.287685 punch from 0x18776a to 0x194292 (0xcb29 bytes) 000058[1] 1788535173.396464 mapread 0x171000 thru 0x17265d (0x165e bytes) 000059[1] 1788535173.469717 punch from 0x2bdb3a to 0x2c9cce (0xc195 bytes) 000060[0] 1788535173.516627 read_OD 0x015000 thru 0x01ffff (0xb000 bytes) 000061[0] 1788535173.580059 mapwrite 0x089000 thru 0x09376d (0xa76e bytes) 000062[0] 1788535173.717388 read_OD 0x137000 thru 0x13cfff (0x6000 bytes) 000064[0] 1788535173.743101 write_OD 0x977000 thru 0x984fff (0xe000 bytes) 000065[0] 1788535173.820161 trunc from 0x985000 to 0x172a27 (0x8125d8 bytes) 000066[0] 1788535173.869168 read_OD 0x140000 thru 0x144fff (0x5000 bytes) 000067[0] 1788535173.912062 read_OD 0x0a3000 thru 0x0a9fff (0x7000 bytes) 000068[1] 1788535173.930338 write_OD 0x77f000 thru 0x780fff (0x2000 bytes) 000069[0] 1788535173.988573 punch from 0x5ee95e to 0x5faa34 (0xc0d7 bytes) 000070[0] 1788535174.016523 mapwrite 0x21d000 thru 0x228675 (0xb676 bytes) 000071[0] 1788535174.205568 mapread 0x01a000 thru 0x01f68d (0x568e bytes) 000072[1] 1788535174.206752 punch from 0x68d408 to 0x690edf (0x3ad8 bytes) 000075[0] 1788535174.251747 mapread 0x6a5000 thru 0x6ac398 (0x7399 bytes) 000077[0] 1788535174.358943 mapread 0x192000 thru 0x19b27b (0x927c bytes) 000081[0] 1788535174.480370 read_OD 0x2c3000 thru 0x2cffff (0xd000 bytes) 000082[0] 1788535174.496296 mapwrite 0x6c0000 thru 0x6c81e0 (0x81e1 bytes) 000084[0] 1788535174.688318 trunc from 0x781000 to 0x492428 (0x2eebd7 bytes) 000087[0] 1788535174.723085 read_OD 0x453000 thru 0x45cfff (0xa000 bytes) 000088[1] 1788535174.749575 write_OD 0x945000 thru 0x948fff (0x4000 bytes) 000091[1] 1788535174.805787 trunc from 0x949000 to 0x646d2f (0x3022d0 bytes) 000092[1] 1788535174.839618 trunc from 0x646d30 to 0x6d43c6 (0x8d697 bytes) 000093[0] 1788535174.852729 punch from 0x033247 to 0x040d85 (0xdb3f bytes) 000094[1] 1788535174.876437 read_OD 0x539000 thru 0x53bfff (0x3000 bytes) 000095[1] 1788535174.901978 punch from 0x45f15a to 0x464d65 (0x5c0c bytes) 000096[1] 1788535174.931184 read_OD 0x0df000 thru 0x0ecfff (0xe000 bytes) 000097[1] 1788535174.948150 write_OD 0x865000 thru 0x86bfff (0x7000 bytes) 000098[1] 1788535174.987410 mapwrite 0x6b7000 thru 0x6b702d (0x2e bytes) 000102[0] 1788535175.112248 write_OD 0x0ee000 thru 0x0f2fff (0x5000 bytes) 000103[1] 1788535175.183259 write_OD 0x6c1000 thru 0x6c4fff (0x4000 bytes) 000104[1] 1788535175.242413 punch from 0x15b570 to 0x15de2e (0x28bf bytes) 000105[0] 1788535175.277501 punch from 0x79205b to 0x79b4ce (0x9474 bytes) 000107[0] 1788535175.316762 mapread 0x4b5000 thru 0x4c1d71 (0xcd72 bytes) 000108[0] 1788535175.347029 mapwrite 0x70a000 thru 0x71393c (0x993d bytes) 000109[1] 1788535175.453259 trunc from 0x86c000 to 0x607aa9 (0x264556 bytes) 000110[0] 1788535175.483806 trunc from 0x607aaa to 0x601064 (0x6a45 bytes) 000112[0] 1788535175.537832 mapwrite 0x964000 thru 0x96d736 (0x9737 bytes) 000114[1] 1788535175.624343 trunc from 0x96d737 to 0x70a17d (0x2635b9 bytes) 000116[1] 1788535175.688755 mapwrite 0x7c2000 thru 0x7c926a (0x726b bytes) 000118[1] 1788535175.867189 read_OD 0x4fd000 thru 0x508fff (0xc000 bytes) 000121[0] 1788535175.924374 mapwrite 0x1bd000 thru 0x1c562e (0x862f bytes) 000123[1] 1788535176.065477 read_OD 0x0cd000 thru 0x0dbfff (0xf000 bytes) 000125[1] 1788535176.151117 punch from 0x1277aa to 0x129c34 (0x248b bytes) 000129[1] 1788535176.229652 read_OD 0x3c2000 thru 0x3cafff (0x9000 bytes) 000130[1] 1788535176.277302 mapwrite 0x82f000 thru 0x83e3ca (0xf3cb bytes) 000131[1] 1788535176.434596 punch from 0x53ec26 to 0x5473b7 (0x8792 bytes) 000132[1] 1788535176.456320 punch from 0x1ee4e7 to 0x1f1334 (0x2e4e bytes) 000135[0] 1788535176.495984 punch from 0x64df22 to 0x64f69c (0x177b bytes) 000136[0] 1788535176.524888 read_OD 0x448000 thru 0x451fff (0xa000 bytes) 000141[0] 1788535176.550681 mapread 0x736000 thru 0x7424b7 (0xc4b8 bytes) 000143[1] 1788535176.588323 punch from 0x07d490 to 0x08ac1c (0xd78d bytes) 000145[0] 1788535176.630359 mapwrite 0x69e000 thru 0x6a23ed (0x43ee bytes) 000148[1] 1788535176.726688 read_OD 0x3e1000 thru 0x3eafff (0xa000 bytes) 000150[0] 1788535176.765056 punch from 0x5dcce5 to 0x5e2bf1 (0x5f0d bytes) 000151[0] 1788535176.797329 write_OD 0x17f000 thru 0x17ffff (0x1000 bytes) 000152[0] 1788535176.846662 mapwrite 0x1fc000 thru 0x206a7e (0xaa7f bytes) 000153[0] 1788535176.933113 punch from 0x712b33 to 0x713c26 (0x10f4 bytes) 000157[1] 1788535176.963839 punch from 0x65664b to 0x6569a3 (0x359 bytes) 000158[1] 1788535176.981722 mapwrite 0x42f000 thru 0x432892 (0x3893 bytes) 000159[0] 1788535177.110487 write_OD 0x223000 thru 0x224fff (0x2000 bytes) 000160[0] 1788535177.173696 trunc from 0x83e3cb to 0x4a4cc0 (0x39970a bytes) 000162[1] 1788535177.320882 trunc from 0x4a4cc1 to 0x7b77b6 (0x312af6 bytes) 000163[0] 1788535177.392513 trunc from 0x7b77b7 to 0x5c1460 (0x1f6356 bytes) 000165[0] 1788535177.493568 mapwrite 0x361000 thru 0x36e9ef (0xd9f0 bytes) 000166[1] 1788535177.632429 write_OD 0x202000 thru 0x20dfff (0xc000 bytes) 000167[0] 1788535177.732752 write_OD 0x8bc000 thru 0x8c8fff (0xd000 bytes) 000168[0] 1788535177.822333 trunc from 0x8c9000 to 0x5ce848 (0x2fa7b7 bytes) 000169[1] 1788535177.855566 write_OD 0x0e7000 thru 0x0e8fff (0x2000 bytes) 000171[1] 1788535177.912698 write_OD 0x425000 thru 0x426fff (0x2000 bytes) 000172[0] 1788535177.959597 punch from 0x1624ab to 0x168333 (0x5e89 bytes) 000173[0] 1788535178.013384 punch from 0x2745d7 to 0x284298 (0xfcc2 bytes) 000177[0] 1788535178.120415 mapwrite 0x19f000 thru 0x1a25d7 (0x35d8 bytes) 000180[0] 1788535178.217993 mapwrite 0x31d000 thru 0x31dc26 (0xc27 bytes) 000181[1] 1788535178.305592 trunc from 0x5ce849 to 0x28e89c (0x33ffac bytes) 000182[0] 1788535178.357372 trunc from 0x28e89d to 0x4d6d13 (0x248477 bytes) 000184[1] 1788535178.403576 mapwrite 0x45c000 thru 0x4621da (0x61db bytes) 000187[1] 1788535178.573236 read_OD 0x12a000 thru 0x12bfff (0x2000 bytes) 000188[0] 1788535178.590397 read_OD 0x476000 thru 0x484fff (0xf000 bytes) 000190[0] 1788535178.608600 write_OD 0x55b000 thru 0x566fff (0xc000 bytes) 000194[1] 1788535178.666793 mapread 0x001000 thru 0x0085e4 (0x75e5 bytes) 000195[1] 1788535178.693356 mapread 0x05c000 thru 0x061e4c (0x5e4d bytes) 000196[0] 1788535178.716027 punch from 0x37a311 to 0x383a7b (0x976b bytes) 000199[1] 1788535178.796636 mapread 0x165000 thru 0x16fdd0 (0xadd1 bytes) 000200[1] 1788535178.871511 write_OD 0x111000 thru 0x112fff (0x2000 bytes) 000202[0] 1788535179.007467 mapread 0x44f000 thru 0x45512d (0x612e bytes) 000203[1] 1788535179.048218 mapwrite 0x1b4000 thru 0x1b5ca7 (0x1ca8 bytes) 000206[1] 1788535179.285179 punch from 0x4efd8b to 0x4fa761 (0xa9d7 bytes) 000207[0] 1788535179.295034 mapread 0x428000 thru 0x436635 (0xe636 bytes) 000208[0] 1788535179.377191 mapread 0x3fd000 thru 0x40b34f (0xe350 bytes) truncating to largest ever: 0x970930 000210[0] 1788535179.408513 trunc from 0x567000 to 0x97092f (0x409930 bytes) 000212[0] 1788535179.482648 trunc from 0x970930 to 0x3bc3ea (0x5b4545 bytes) 000214[0] 1788535179.555457 trunc from 0x3bc3eb to 0x08804c (0x33439e bytes) 000216[0] 1788535179.624204 mapwrite 0x7b2000 thru 0x7b3841 (0x1842 bytes) 000217[0] 1788535179.809818 mapread 0x0f6000 thru 0x105acb (0xfacc bytes) 000218[1] 1788535179.871730 trunc from 0x7b3842 to 0x2c151a (0x4f2327 bytes) 000219[1] 1788535179.935474 trunc from 0x2c151b to 0x84024d (0x57ed33 bytes) 000220[0] 1788535180.005775 punch from 0x19d347 to 0x1ac96a (0xf624 bytes) 000221[1] 1788535180.038395 read_OD 0x2ad000 thru 0x2b9fff (0xd000 bytes) 000223[1] 1788535180.081418 trunc from 0x84024e to 0x12e5fe (0x711c4f bytes) 000226[0] 1788535180.167085 trunc from 0x12e5ff to 0x595e9f (0x4678a1 bytes) 000227[1] 1788535180.238963 write_OD 0x021000 thru 0x024fff (0x4000 bytes) 000228[1] 1788535180.355019 mapread 0x27b000 thru 0x27cd86 (0x1d87 bytes) 000231[1] 1788535180.378259 write_OD 0x04e000 thru 0x04ffff (0x2000 bytes) 000232[1] 1788535180.449114 mapwrite 0x8fb000 thru 0x8ff1d2 (0x41d3 bytes) 000233[0] 1788535180.581142 mapread 0x09d000 thru 0x0a7b6f (0xab70 bytes) 000234[1] 1788535180.640905 punch from 0x372f3c to 0x376bf8 (0x3cbd bytes) 000236[0] 1788535180.690148 punch from 0x16ff31 to 0x17bfab (0xc07b bytes) 000238[0] 1788535180.733497 mapread 0x07b000 thru 0x089c16 (0xec17 bytes) 000239[0] 1788535180.785565 mapread 0x4a1000 thru 0x4a6fca (0x5fcb bytes) 000240[1] 1788535180.841059 write_OD 0x6ad000 thru 0x6aefff (0x2000 bytes) 000241[0] 1788535180.985374 punch from 0x1dc9a3 to 0x1e9816 (0xce74 bytes) 000242[0] 1788535181.043338 read_OD 0x758000 thru 0x763fff (0xc000 bytes) 000243[0] 1788535181.098611 punch from 0x3d66d9 to 0x3e51f1 (0xeb19 bytes) 000244[0] 1788535181.142439 mapread 0x46d000 thru 0x47a635 (0xd636 bytes) 000245[1] 1788535181.181322 mapread 0x60b000 thru 0x615676 (0xa677 bytes) 000246[1] 1788535181.235000 mapread 0x0d6000 thru 0x0e058c (0xa58d bytes) 000248[0] 1788535181.261388 write_OD 0x99e000 thru 0x9a6fff (0x9000 bytes) 000252[0] 1788535181.375846 punch from 0x183125 to 0x186cdd (0x3bb9 bytes) 000253[1] 1788535181.434121 mapwrite 0x2c5000 thru 0x2cbaf6 (0x6af7 bytes) 000257[1] 1788535181.607718 write_OD 0x83a000 thru 0x83ffff (0x6000 bytes) 000259[1] 1788535181.723138 punch from 0x56c0b7 to 0x57a44a (0xe394 bytes) 000260[0] 1788535181.779774 mapread 0x096000 thru 0x09a856 (0x4857 bytes) 000262[1] 1788535181.871193 trunc from 0x9a7000 to 0x73760e (0x26f9f1 bytes) 000263[1] 1788535181.928661 punch from 0x02956f to 0x02a672 (0x1104 bytes) 000264[1] 1788535181.964059 trunc from 0x73760f to 0x5f1d72 (0x14589c bytes) 000265[1] 1788535182.000307 read_OD 0x20a000 thru 0x20bfff (0x2000 bytes) 000267[0] 1788535182.029839 trunc from 0x5f1d73 to 0x338b08 (0x2b926a bytes) 000269[0] 1788535182.129941 punch from 0x2077f3 to 0x210f18 (0x9726 bytes) 000270[0] 1788535182.160553 write_OD 0x2e8000 thru 0x2edfff (0x6000 bytes) 000273[1] 1788535182.257163 trunc from 0x338b09 to 0x3e5ec0 (0xad3b8 bytes) 000274[1] 1788535182.332504 write_OD 0x3b6000 thru 0x3bafff (0x5000 bytes) 000276[1] 1788535182.391420 read_OD 0x1c1000 thru 0x1ccfff (0xc000 bytes) 000277[1] 1788535182.400720 mapwrite 0x330000 thru 0x332e80 (0x2e81 bytes) 000280[0] 1788535182.583178 write_OD 0x705000 thru 0x706fff (0x2000 bytes) 000284[1] 1788535182.685009 write_OD 0x801000 thru 0x80cfff (0xc000 bytes) 000285[1] 1788535182.782579 trunc from 0x80d000 to 0x7c7090 (0x45f6f bytes) 000288[1] 1788535182.880493 write_OD 0x63c000 thru 0x63efff (0x3000 bytes) 000289[0] 1788535182.946211 mapwrite 0x1a9000 thru 0x1b1800 (0x8801 bytes) 000290[1] 1788535183.070731 write_OD 0x6e4000 thru 0x6e6fff (0x3000 bytes) truncating to largest ever: 0x9fd847 000291[1] 1788535183.180497 trunc from 0x7c7091 to 0x9fd846 (0x2367b6 bytes) 000292[1] 1788535183.234263 mapwrite 0x7ab000 thru 0x7b5604 (0xa605 bytes) 000294[1] 1788535183.471992 mapwrite 0x162000 thru 0x1653e1 (0x33e2 bytes) 000295[0] 1788535183.642346 punch from 0x686719 to 0x68ba42 (0x532a bytes) 000297[1] 1788535183.762532 mapread 0x883000 thru 0x88772f (0x4730 bytes) 000299[0] 1788535183.858660 mapwrite 0x99f000 thru 0x9a4f32 (0x5f33 bytes) 000300[0] 1788535184.018767 mapread 0x6f9000 thru 0x6fba27 (0x2a28 bytes) 000301[1] 1788535184.057465 punch from 0x0ef10f to 0x0f1edf (0x2dd1 bytes) 000304[1] 1788535184.076648 punch from 0x7b2725 to 0x7c2320 (0xfbfc bytes) 000305[1] 1788535184.096710 write_OD 0x30c000 thru 0x31afff (0xf000 bytes) 000306[1] 1788535184.205319 write_OD 0x22d000 thru 0x234fff (0x8000 bytes) 000308[1] 1788535184.264204 read_OD 0x0a3000 thru 0x0aefff (0xc000 bytes) 000311[1] 1788535184.282257 read_OD 0x339000 thru 0x345fff (0xd000 bytes) 000312[1] 1788535184.297305 punch from 0x160434 to 0x16b2d2 (0xae9f bytes) 000313[0] 1788535184.336851 punch from 0x1dab51 to 0x1de6b7 (0x3b67 bytes) 000314[0] 1788535184.381639 read_OD 0x1d6000 thru 0x1e0fff (0xb000 bytes) 000315[1] 1788535184.405358 trunc from 0x9fd847 to 0x7c3b0d (0x239d39 bytes) 000316[0] 1788535184.465364 write_OD 0x9f9000 thru 0x9fefff (0x6000 bytes) 000317[1] 1788535184.535814 punch from 0x183b93 to 0x18b56d (0x79db bytes) 000318[0] 1788535184.555747 mapwrite 0x95f000 thru 0x96c91d (0xd91e bytes) 000320[0] 1788535184.714242 mapwrite 0x4c1000 thru 0x4cbf94 (0xaf95 bytes) 000321[0] 1788535184.857291 mapread 0x668000 thru 0x67664d (0xe64e bytes) 000322[1] 1788535184.936652 trunc from 0x9ff000 to 0x0d69da (0x928625 bytes) 000324[1] 1788535185.014115 read_OD 0x013000 thru 0x021fff (0xf000 bytes) 000325[1] 1788535185.049677 trunc from 0x0d69db to 0x35b399 (0x2849bf bytes) 000326[0] 1788535185.086525 read_OD 0x2cb000 thru 0x2d3fff (0x9000 bytes) 000330[0] 1788535185.134210 mapwrite 0x961000 thru 0x9676df (0x66e0 bytes) 000332[0] 1788535185.461572 punch from 0x11beff to 0x11d4ed (0x15ef bytes) 000333[0] 1788535185.489895 mapread 0x154000 thru 0x1612b8 (0xd2b9 bytes) 000336[1] 1788535185.512192 write_OD 0x2bf000 thru 0x2cbfff (0xd000 bytes) 000338[0] 1788535185.581610 trunc from 0x9676e0 to 0x943e19 (0x238c6 bytes) 000339[1] 1788535185.659193 mapwrite 0x760000 thru 0x76f832 (0xf833 bytes) 000341[0] 1788535186.042905 mapwrite 0x7a3000 thru 0x7a30e1 (0xe2 bytes) 000343[0] 1788535186.187980 punch from 0x3520c7 to 0x3547e6 (0x2720 bytes) 000344[0] 1788535186.194160 punch from 0x570216 to 0x57c5fa (0xc3e5 bytes) 000345[0] 1788535186.199308 read_OD 0x3c8000 thru 0x3cbfff (0x4000 bytes) 000346[1] 1788535186.227114 read_OD 0x083000 thru 0x088fff (0x6000 bytes) 000349[1] 1788535186.257566 write_OD 0x9f5000 thru 0x9fcfff (0x8000 bytes) 000350[1] 1788535186.330996 read_OD 0x6f1000 thru 0x6f9fff (0x9000 bytes) 000351[0] 1788535186.344667 mapwrite 0x1a4000 thru 0x1b0279 (0xc27a bytes) 000352[1] 1788535186.518311 mapwrite 0x15b000 thru 0x168826 (0xd827 bytes) 000354[0] 1788535186.662382 mapwrite 0x22c000 thru 0x22e882 (0x2883 bytes) 000355[0] 1788535186.834747 trunc from 0x9fd000 to 0x828bd6 (0x1d4429 bytes) 000356[0] 1788535186.886411 write_OD 0x671000 thru 0x673fff (0x3000 bytes) 000359[0] 1788535186.965085 mapwrite 0x496000 thru 0x49f8b4 (0x98b5 bytes) 000363[0] 1788535187.149785 write_OD 0x978000 thru 0x97afff (0x3000 bytes) 000367[1] 1788535187.271633 mapwrite 0x10e000 thru 0x117c49 (0x9c4a bytes) 000368[0] 1788535187.487426 read_OD 0x303000 thru 0x304fff (0x2000 bytes) 000369[0] 1788535187.538351 read_OD 0x67c000 thru 0x687fff (0xc000 bytes) 000370[0] 1788535187.547247 mapwrite 0x9ed000 thru 0x9f9334 (0xc335 bytes) 000371[0] 1788535187.766667 mapwrite 0x638000 thru 0x6450f0 (0xd0f1 bytes) 000372[0] 1788535187.915410 trunc from 0x9f9335 to 0x61b81e (0x3ddb16 bytes) 000374[1] 1788535187.990516 punch from 0x3e7bd4 to 0x3f4424 (0xc851 bytes) 000375[1] 1788535188.044238 read_OD 0x606000 thru 0x606fff (0x1000 bytes) 000376[1] 1788535188.060310 mapwrite 0x0e8000 thru 0x0f5dd3 (0xddd4 bytes) 000381[1] 1788535188.222639 punch from 0x0b4928 to 0x0b8a95 (0x416e bytes) 000382[1] 1788535188.249112 punch from 0x027c57 to 0x02bb2e (0x3ed8 bytes) 000383[0] 1788535188.256437 mapwrite 0x5ac000 thru 0x5b8ac8 (0xcac9 bytes) 000384[1] 1788535188.360502 punch from 0x4c1ae1 to 0x4c4b10 (0x3030 bytes) 000385[0] 1788535188.393110 trunc from 0x61b81f to 0x9b5335 (0x399b17 bytes) 000387[0] 1788535188.478131 mapwrite 0x7ae000 thru 0x7bbe63 (0xde64 bytes) 000389[0] 1788535188.579304 read_OD 0x238000 thru 0x244fff (0xd000 bytes) 000391[0] 1788535188.606244 punch from 0x50056d to 0x50f88d (0xf321 bytes) 000392[1] 1788535188.645331 write_OD 0x144000 thru 0x147fff (0x4000 bytes) 000393[1] 1788535188.710226 mapread 0x0d3000 thru 0x0d33a6 (0x3a7 bytes) 000394[0] 1788535188.740426 read_OD 0x8e8000 thru 0x8eafff (0x3000 bytes) 000396[1] 1788535188.754466 mapread 0x053000 thru 0x05e4bd (0xb4be bytes) 000398[1] 1788535188.833497 mapwrite 0x51f000 thru 0x526231 (0x7232 bytes) 000400[0] 1788535189.045372 trunc from 0x9b5336 to 0x3e648f (0x5ceea6 bytes) 000402[1] 1788535189.119228 mapwrite 0x420000 thru 0x4294c4 (0x94c5 bytes) 000403[1] 1788535189.311240 write_OD 0x66c000 thru 0x675fff (0xa000 bytes) 000404[1] 1788535189.424131 mapwrite 0x7dd000 thru 0x7e0db6 (0x3db7 bytes) 000405[1] 1788535189.589289 trunc from 0x7e0db7 to 0x659432 (0x187984 bytes) 000406[1] 1788535189.623919 punch from 0x35d60c to 0x3688c5 (0xb2ba bytes) 000407[1] 1788535189.641367 punch from 0x605a08 to 0x60ce87 (0x7480 bytes) 000410[0] 1788535189.663903 trunc from 0x659433 to 0x62bcb1 (0x2d781 bytes) 000411[0] 1788535189.739304 write_OD 0x86d000 thru 0x86ffff (0x3000 bytes) 000413[1] 1788535189.930215 mapwrite 0x5cb000 thru 0x5d4b9a (0x9b9b bytes) 000414[1] 1788535190.109849 mapwrite 0x7a1000 thru 0x7a3500 (0x2501 bytes) 000416[0] 1788535190.227531 write_OD 0x654000 thru 0x658fff (0x5000 bytes) 000417[0] 1788535190.319984 read_OD 0x663000 thru 0x666fff (0x4000 bytes) 000418[1] 1788535190.343197 trunc from 0x870000 to 0x64ed85 (0x22127a bytes) 000420[1] 1788535190.430066 mapread 0x465000 thru 0x46a9c7 (0x59c8 bytes) 000422[0] 1788535190.456747 read_OD 0x1af000 thru 0x1bdfff (0xf000 bytes) 000423[1] 1788535190.499980 mapwrite 0x1b6000 thru 0x1c24b9 (0xc4ba bytes) 000425[0] 1788535190.725296 mapread 0x06d000 thru 0x075b64 (0x8b65 bytes) 000426[0] 1788535190.784659 punch from 0x2e1e91 to 0x2ec7d1 (0xa941 bytes) 000427[1] 1788535190.813393 mapread 0x35e000 thru 0x366954 (0x8955 bytes) 000428[1] 1788535190.864923 read_OD 0x3b7000 thru 0x3bdfff (0x7000 bytes) 000432[1] 1788535190.894792 write_OD 0x8c2000 thru 0x8c5fff (0x4000 bytes) 000433[1] 1788535190.957523 mapwrite 0x7fd000 thru 0x8023a3 (0x53a4 bytes) 000434[0] 1788535191.100079 punch from 0x8bdbb2 to 0x8bea15 (0xe64 bytes) 000436[1] 1788535191.129720 read_OD 0x0c3000 thru 0x0c6fff (0x4000 bytes) 000440[1] 1788535191.171111 mapread 0x607000 thru 0x610218 (0x9219 bytes) 000442[0] 1788535191.198232 mapwrite 0x097000 thru 0x09fa23 (0x8a24 bytes) 000443[0] 1788535191.388659 punch from 0x3251d0 to 0x326a5b (0x188c bytes) 000444[1] 1788535191.407471 mapwrite 0x2a8000 thru 0x2b2ea6 (0xaea7 bytes) 000446[1] 1788535191.517080 trunc from 0x8c6000 to 0x70a43f (0x1bbbc0 bytes) 000447[0] 1788535191.543990 punch from 0x5e4d9a to 0x5e83a2 (0x3609 bytes) 000450[1] 1788535191.578293 write_OD 0x3b2000 thru 0x3c0fff (0xf000 bytes) 000451[1] 1788535191.637772 trunc from 0x70a440 to 0x66fe99 (0x9a5a6 bytes) 000453[1] 1788535191.703601 mapread 0x051000 thru 0x059a21 (0x8a22 bytes) 000455[1] 1788535191.743197 write_OD 0x601000 thru 0x607fff (0x7000 bytes) 000456[0] 1788535191.788136 punch from 0x57503e to 0x57cbdc (0x7b9f bytes) 000457[1] 1788535191.802570 write_OD 0x456000 thru 0x45efff (0x9000 bytes) 000460[1] 1788535191.848514 mapread 0x098000 thru 0x09f180 (0x7181 bytes) 000464[0] 1788535191.859673 read_OD 0x522000 thru 0x529fff (0x8000 bytes) 000466[1] 1788535191.868587 trunc from 0x66fe9a to 0x3e03de (0x28fabb bytes) 000467[0] 1788535191.899264 mapwrite 0x461000 thru 0x467e09 (0x6e0a bytes) 000468[0] 1788535192.035412 punch from 0x340e24 to 0x344da7 (0x3f84 bytes) 000471[0] 1788535192.067409 mapwrite 0x1f8000 thru 0x1fb4be (0x34bf bytes) 000473[0] 1788535192.198686 trunc from 0x467e0a to 0x354006 (0x113e03 bytes) 000475[1] 1788535192.251616 write_OD 0x3a5000 thru 0x3affff (0xb000 bytes) 000477[0] 1788535192.305325 trunc from 0x3b0000 to 0x337785 (0x7887a bytes) 000482[1] 1788535192.407465 punch from 0x262e83 to 0x270b0e (0xdc8c bytes) 000485[1] 1788535192.436496 read_OD 0x2bb000 thru 0x2bcfff (0x2000 bytes) 000486[1] 1788535192.470211 read_OD 0x30a000 thru 0x317fff (0xe000 bytes) 000490[1] 1788535192.507504 mapread 0x32a000 thru 0x33693c (0xc93d bytes) 000491[0] 1788535192.600396 trunc from 0x337786 to 0x782ee6 (0x44b761 bytes) 000492[1] 1788535192.646136 mapread 0x0b7000 thru 0x0bfc24 (0x8c25 bytes) 000493[1] 1788535192.729927 write_OD 0x22c000 thru 0x22cfff (0x1000 bytes) 000497[1] 1788535192.793001 mapwrite 0x145000 thru 0x147975 (0x2976 bytes) 000498[1] 1788535193.028587 write_OD 0x312000 thru 0x31dfff (0xc000 bytes) All operations completed A-OK! PASS 16a (60s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 11:19:59 (1788535199) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.613045 s, 13.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.464404 s, 18.1 MB/s Chance of close/open is 1 in 50 Seed set to 5203 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole skipping zero size read skipping zero length punch hole skipping zero length punch hole skipping zero length punch hole skipping zero size read skipping zero size read skipping zero length punch hole 000012[0] 1788535204.928421 mapwrite 0x8e09b9 thru 0x8e19b0 (0xff8 bytes) 000013[0] 1788535205.089267 mapwrite 0x6c2d00 thru 0x6c3173 (0x474 bytes) 000014[1] 1788535205.197436 punch from 0x8809b7 to 0x881af8 (0x1142 bytes) 000015[0] 1788535205.231184 read 0x32eec2 thru 0x32f43b (0x57a bytes) 000016[1] 1788535205.257289 mapwrite 0x086b3a thru 0x087b5e (0x1025 bytes) 000017[0] 1788535205.379153 mapwrite 0x043bdd thru 0x044826 (0xc4a bytes) 000018[1] 1788535205.479389 punch from 0x6f2861 to 0x6f3dd4 (0x1574 bytes) 000019[0] 1788535205.511774 read 0x89f62b thru 0x8a02c9 (0xc9f bytes) 000020[1] 1788535205.535369 mapread 0x6f0384 thru 0x6f12aa (0xf27 bytes) truncating to largest ever: 0xbefca 000021[0] 1788535205.583191 trunc from 0x8e19b1 to 0x0befc9 (0x8229e7 bytes) 000022[1] 1788535205.670476 read 0x09222c thru 0x092e79 (0xc4e bytes) 000023[1] 1788535205.732911 punch from 0x07f80c to 0x080a68 (0x125d bytes) 000024[0] 1788535205.835388 read 0x07b6e7 thru 0x07ba9c (0x3b6 bytes) 000025[1] 1788535205.928196 mapwrite 0x08446b thru 0x0861b4 (0x1d4a bytes) 000026[1] 1788535206.154644 write 0x4b9e37 thru 0x4bb67e (0x1848 bytes) 000027[0] 1788535206.201138 write 0x64b71b thru 0x64c3d6 (0xcbc bytes) 000028[1] 1788535206.240498 mapread 0x4c68f9 thru 0x4c7662 (0xd6a bytes) 000030[0] 1788535206.318123 mapread 0x58cc4d thru 0x58ccce (0x82 bytes) truncating to largest ever: 0x85807c 000033[1] 1788535206.360417 trunc from 0x64c3d7 to 0x85807b (0x20bca5 bytes) 000035[1] 1788535206.431367 mapwrite 0x9f995d thru 0x9f9da4 (0x448 bytes) 000039[1] 1788535206.518140 punch from 0x7b3739 to 0x7b4e13 (0x16db bytes) 000040[1] 1788535206.523275 mapread 0x5a4822 thru 0x5a6285 (0x1a64 bytes) 000043[1] 1788535206.598411 trunc from 0x9f9da5 to 0x3cfe99 (0x629f0b bytes) 000044[1] 1788535206.623208 read 0x0587be thru 0x058ef3 (0x736 bytes) 000046[1] 1788535206.768077 read 0x111f5e thru 0x112a67 (0xb0a bytes) 000047[0] 1788535206.807615 mapwrite 0x953b69 thru 0x9553aa (0x1842 bytes) 000048[1] 1788535207.144904 mapread 0x661baa thru 0x662813 (0xc6a bytes) 000053[1] 1788535207.241226 mapread 0x051bd2 thru 0x0532a9 (0x16d8 bytes) truncating to largest ever: 0x87b35e 000054[0] 1788535207.351717 trunc from 0x9553ab to 0x87b35d (0xda04d bytes) 000056[0] 1788535207.564102 read 0x3d7c7e thru 0x3d8c28 (0xfab bytes) 000060[0] 1788535207.733255 mapwrite 0x8bbfb4 thru 0x8bc906 (0x953 bytes) 000063[0] 1788535208.043509 punch from 0x114b44 to 0x115e10 (0x12cd bytes) 000064[1] 1788535208.072485 punch from 0x525675 to 0x5273df (0x1d6b bytes) 000069[1] 1788535208.243019 punch from 0x5606f2 to 0x562551 (0x1e60 bytes) 000072[1] 1788535208.311899 trunc from 0x8bc907 to 0x486c9b (0x435c6b bytes) 000073[1] 1788535208.388830 trunc from 0x486c9c to 0x02946b (0x45d830 bytes) 000075[1] 1788535208.416725 trunc from 0x02946c to 0x87552e (0x84c0c3 bytes) 000079[1] 1788535208.488540 mapread 0x739973 thru 0x73b3e2 (0x1a70 bytes) 000080[0] 1788535208.613743 trunc from 0x87552f to 0x74c79e (0x128d90 bytes) 000081[1] 1788535208.769905 mapwrite 0x6257de thru 0x62710f (0x1932 bytes) 000082[1] 1788535208.910709 mapread 0x6f6512 thru 0x6f6547 (0x36 bytes) 000084[0] 1788535208.970278 mapread 0x2e9ee1 thru 0x2e9f17 (0x37 bytes) 000086[1] 1788535209.032606 punch from 0x316469 to 0x31714d (0xce5 bytes) 000087[1] 1788535209.085548 mapwrite 0x4166f6 thru 0x41711c (0xa27 bytes) 000088[0] 1788535209.210089 write 0x6ea51b thru 0x6eb828 (0x130e bytes) 000091[1] 1788535209.331034 mapwrite 0x698ad9 thru 0x698b57 (0x7f bytes) 000092[1] 1788535209.501955 mapread 0x636e4c thru 0x6385f0 (0x17a5 bytes) 000093[1] 1788535209.563751 mapwrite 0x797e9c thru 0x7989b0 (0xb15 bytes) 000094[1] 1788535209.697806 mapwrite 0x115635 thru 0x116346 (0xd12 bytes) 000095[1] 1788535209.757031 trunc from 0x7989b1 to 0x69b888 (0xfd128 bytes) 000096[0] 1788535209.805584 trunc from 0x69b889 to 0x109a42 (0x591e46 bytes) 000097[1] 1788535209.856162 read 0x0051c1 thru 0x006501 (0x1341 bytes) 000098[0] 1788535209.880314 mapread 0x0da9d4 thru 0x0dc349 (0x1976 bytes) 000101[0] 1788535209.938293 mapwrite 0x84aa29 thru 0x84ae30 (0x408 bytes) 000104[1] 1788535210.057773 punch from 0x10ca8a to 0x10e4d0 (0x1a47 bytes) 000107[0] 1788535210.143386 punch from 0x6015f4 to 0x602aac (0x14b9 bytes) 000109[0] 1788535210.212402 trunc from 0x84ae31 to 0x22b159 (0x61fcd7 bytes) 000111[1] 1788535210.340734 write 0x59f3f5 thru 0x5a0c59 (0x1865 bytes) 000112[0] 1788535210.431389 read 0x0f1051 thru 0x0f20e5 (0x1095 bytes) 000113[0] 1788535210.535379 punch from 0x36a348 to 0x36af1a (0xbd3 bytes) 000114[1] 1788535210.557689 mapwrite 0x7faad1 thru 0x7fae98 (0x3c8 bytes) 000115[1] 1788535210.744440 punch from 0x696e3c to 0x6985e2 (0x17a7 bytes) 000116[1] 1788535210.776616 mapread 0x66e7e2 thru 0x66f78a (0xfa9 bytes) 000119[1] 1788535210.791542 read 0x5f64a1 thru 0x5f6e30 (0x990 bytes) 000120[0] 1788535210.818710 write 0x3bb33d thru 0x3bbe9b (0xb5f bytes) 000121[1] 1788535210.900425 read 0x47c991 thru 0x47cb71 (0x1e1 bytes) 000122[0] 1788535210.948589 read 0x434520 thru 0x435e2b (0x190c bytes) 000124[0] 1788535211.015172 write 0x1a7a9b thru 0x1a9097 (0x15fd bytes) 000126[0] 1788535211.109447 write 0x8d27d2 thru 0x8d41c4 (0x19f3 bytes) 000128[1] 1788535211.131475 write 0x0faa66 thru 0x0fb391 (0x92c bytes) 000129[1] 1788535211.189074 mapwrite 0x7415f2 thru 0x7426a0 (0x10af bytes) 000131[1] 1788535211.336637 write 0x9ff5fd thru 0x9fffff (0xa03 bytes) 000132[1] 1788535211.372813 mapwrite 0x620247 thru 0x6220e3 (0x1e9d bytes) 000135[1] 1788535211.577267 write 0x1a63c6 thru 0x1a7068 (0xca3 bytes) 000136[0] 1788535211.592533 mapread 0x3a64aa thru 0x3a6cd7 (0x82e bytes) 000137[0] 1788535211.658966 punch from 0x23a44c to 0x23b2c0 (0xe75 bytes) 000138[0] 1788535211.706425 write 0x677e1e thru 0x677fcf (0x1b2 bytes) 000143[1] 1788535211.795302 mapread 0x11a857 thru 0x11c4ca (0x1c74 bytes) 000146[0] 1788535211.892552 punch from 0x8db4ba to 0x8dd146 (0x1c8d bytes) 000147[0] 1788535211.970188 write 0x38e762 thru 0x38f3b7 (0xc56 bytes) 000149[0] 1788535212.027822 read 0x06930a thru 0x069c7f (0x976 bytes) 000150[1] 1788535212.085761 read 0x18e282 thru 0x18e676 (0x3f5 bytes) 000151[1] 1788535212.139248 mapread 0x0208ff thru 0x020c51 (0x353 bytes) 000153[0] 1788535212.168223 write 0x8b48af thru 0x8b5308 (0xa5a bytes) 000155[0] 1788535212.343223 write 0x40814b thru 0x40880b (0x6c1 bytes) 000156[0] 1788535212.432095 trunc from 0xa00000 to 0x0407fc (0x9bf803 bytes) 000158[1] 1788535212.539293 mapwrite 0x907e07 thru 0x908a41 (0xc3b bytes) 000161[0] 1788535212.880312 read 0x149266 thru 0x14a65a (0x13f5 bytes) 000162[0] 1788535212.912327 read 0x6266b3 thru 0x627607 (0xf55 bytes) 000163[0] 1788535212.935161 trunc from 0x908a42 to 0x31c293 (0x5ec7ae bytes) 000164[1] 1788535213.018691 punch from 0x2121d8 to 0x21344c (0x1275 bytes) 000166[1] 1788535213.065214 mapread 0x1a9768 thru 0x1ab6c9 (0x1f62 bytes) 000167[1] 1788535213.113291 mapread 0x20a67c thru 0x20ba55 (0x13da bytes) 000168[0] 1788535213.151224 write 0x585eb9 thru 0x586d70 (0xeb8 bytes) 000170[0] 1788535213.184368 punch from 0x223885 to 0x223a03 (0x17f bytes) 000172[1] 1788535213.253685 trunc from 0x586d71 to 0x06e3b6 (0x5189ba bytes) 000173[0] 1788535213.308553 read 0x04ba5c thru 0x04d111 (0x16b6 bytes) truncating to largest ever: 0x8e8afc 000175[0] 1788535213.352764 trunc from 0x06e3b7 to 0x8e8afb (0x87a745 bytes) 000176[1] 1788535213.417362 read 0x485c84 thru 0x487a90 (0x1e0d bytes) 000178[1] 1788535213.476520 mapread 0x72ea81 thru 0x72f406 (0x986 bytes) 000179[0] 1788535213.508245 mapread 0x6e2206 thru 0x6e41f1 (0x1fec bytes) 000185[1] 1788535213.596125 mapread 0x543413 thru 0x5452d6 (0x1ec4 bytes) 000186[1] 1788535213.644577 mapread 0x542a95 thru 0x54333d (0x8a9 bytes) 000187[0] 1788535213.665804 write 0x0c83fd thru 0x0c8440 (0x44 bytes) 000189[0] 1788535213.721929 mapwrite 0x414666 thru 0x415f69 (0x1904 bytes) 000191[0] 1788535213.870331 trunc from 0x8e8afc to 0x60a4ab (0x2de650 bytes) 000195[0] 1788535214.038563 trunc from 0x60a4ac to 0x858346 (0x24de9b bytes) 000196[0] 1788535214.076701 trunc from 0x858347 to 0x8b9579 (0x61233 bytes) 000197[0] 1788535214.168132 write 0x2c9e92 thru 0x2ca60c (0x77b bytes) 000198[1] 1788535214.220498 punch from 0x21c729 to 0x21c8f8 (0x1d0 bytes) 000199[1] 1788535214.301213 mapread 0x7e3ec9 thru 0x7e49b4 (0xaec bytes) 000201[0] 1788535214.518582 trunc from 0x8b957a to 0x14bbae (0x76d9cb bytes) 000202[0] 1788535214.635249 mapread 0x11496e thru 0x115123 (0x7b6 bytes) 000203[1] 1788535214.669581 write 0x534432 thru 0x534cb6 (0x885 bytes) 000204[0] 1788535214.716620 punch from 0x408037 to 0x4098ce (0x1898 bytes) 000205[1] 1788535214.760139 mapread 0x0f0a45 thru 0x0f1962 (0xf1e bytes) 000206[0] 1788535214.817500 mapwrite 0x85d320 thru 0x85d40f (0xf0 bytes) 000207[1] 1788535214.987394 write 0x73ef36 thru 0x740599 (0x1664 bytes) 000208[1] 1788535215.039026 punch from 0x22e3ee to 0x22fbe2 (0x17f5 bytes) 000209[1] 1788535215.092201 mapwrite 0x839a93 thru 0x83b237 (0x17a5 bytes) 000211[1] 1788535215.225346 mapwrite 0x51d785 thru 0x51f681 (0x1efd bytes) 000212[0] 1788535215.332584 write 0x43b09f thru 0x43c40a (0x136c bytes) 000213[1] 1788535215.383435 punch from 0x432ea2 to 0x4334c0 (0x61f bytes) 000214[1] 1788535215.480243 trunc from 0x85d410 to 0x114a82 (0x74898d bytes) 000215[1] 1788535215.524779 punch from 0x08e7f5 to 0x08f813 (0x101f bytes) 000217[1] 1788535215.580221 trunc from 0x114a83 to 0x1abff5 (0x97573 bytes) 000218[1] 1788535215.618274 mapwrite 0x882665 thru 0x88353b (0xed7 bytes) 000219[1] 1788535215.815020 mapwrite 0x9fa861 thru 0x9fc1a2 (0x1942 bytes) 000220[1] 1788535215.985069 write 0x7c27a2 thru 0x7c3be0 (0x143f bytes) 000221[1] 1788535215.995571 mapread 0x61cecd thru 0x61d8cb (0x9ff bytes) 000222[1] 1788535216.046347 mapwrite 0x3a36ec thru 0x3a4044 (0x959 bytes) 000224[0] 1788535216.183412 write 0x40e0c8 thru 0x40ffeb (0x1f24 bytes) 000228[1] 1788535216.305825 read 0x380f90 thru 0x381da7 (0xe18 bytes) 000230[0] 1788535216.353346 read 0x54d907 thru 0x54e140 (0x83a bytes) 000232[1] 1788535216.387693 mapread 0x4f612b thru 0x4f6b28 (0x9fe bytes) 000233[1] 1788535216.415920 punch from 0x9cf2c4 to 0x9d0ff1 (0x1d2e bytes) 000234[0] 1788535216.421466 mapwrite 0x48ebdc thru 0x4903fe (0x1823 bytes) 000238[1] 1788535216.561588 punch from 0x0b9cc3 to 0x0ba87e (0xbbc bytes) 000240[0] 1788535216.607996 trunc from 0x9fc1a3 to 0x8957d3 (0x1669cf bytes) 000241[1] 1788535216.663133 mapwrite 0x389df9 thru 0x38b329 (0x1531 bytes) 000242[0] 1788535216.824345 read 0x6d59af thru 0x6d6f03 (0x1555 bytes) 000243[0] 1788535216.853005 mapread 0x39ce18 thru 0x39e27f (0x1468 bytes) 000245[1] 1788535216.923482 mapread 0x4dac5c thru 0x4db9fb (0xda0 bytes) 000246[0] 1788535216.957345 trunc from 0x8957d4 to 0x1154ff (0x7802d4 bytes) 000247[1] 1788535217.022977 mapwrite 0x90345c thru 0x904fea (0x1b8f bytes) 000248[0] 1788535217.215775 mapwrite 0x6285b0 thru 0x6286ee (0x13f bytes) 000250[1] 1788535217.402283 punch from 0x1637cc to 0x163804 (0x39 bytes) 000251[1] 1788535217.449651 mapwrite 0x0d27a8 thru 0x0d3bdb (0x1434 bytes) 000253[0] 1788535217.546568 trunc from 0x904feb to 0x0eccc3 (0x818327 bytes) 000255[0] 1788535217.652144 mapwrite 0x0b7f22 thru 0x0b8f1f (0xffe bytes) 000257[0] 1788535217.723280 read 0x0447e5 thru 0x045a7e (0x129a bytes) 000258[0] 1788535217.782379 mapwrite 0x45db53 thru 0x45df15 (0x3c3 bytes) 000259[0] 1788535218.001267 read 0x070bed thru 0x070e78 (0x28c bytes) 000260[0] 1788535218.007497 mapread 0x0b9519 thru 0x0babd2 (0x16ba bytes) 000261[1] 1788535218.008324 trunc from 0x45df16 to 0x338f63 (0x124fb2 bytes) 000263[0] 1788535218.193043 mapread 0x13eabc thru 0x13f95f (0xea4 bytes) 000264[0] 1788535218.327622 mapwrite 0x86cce3 thru 0x86daa9 (0xdc7 bytes) 000266[0] 1788535218.605516 trunc from 0x86daaa to 0x4a2d59 (0x3cad50 bytes) 000268[1] 1788535218.740337 read 0x15afb4 thru 0x15bdfb (0xe48 bytes) 000269[1] 1788535218.800735 mapread 0x3c6628 thru 0x3c72a1 (0xc7a bytes) 000270[1] 1788535218.897733 punch from 0x0f0956 to 0x0f148b (0xb36 bytes) 000271[0] 1788535218.954284 write 0x12c7f4 thru 0x12dec9 (0x16d6 bytes) 000273[0] 1788535219.009374 mapread 0x1109c0 thru 0x1114aa (0xaeb bytes) 000274[1] 1788535219.041499 read 0x200187 thru 0x2004a4 (0x31e bytes) 000275[1] 1788535219.090348 write 0x36798e thru 0x3687ff (0xe72 bytes) 000283[1] 1788535219.503204 write 0x6fc953 thru 0x6fdc53 (0x1301 bytes) 000285[1] 1788535219.567308 punch from 0x4c6b62 to 0x4c6be3 (0x82 bytes) 000289[1] 1788535219.671301 punch from 0x2cb816 to 0x2cc45c (0xc47 bytes) 000290[0] 1788535219.728569 write 0x827278 thru 0x828898 (0x1621 bytes) 000291[1] 1788535219.805431 punch from 0x4cdb34 to 0x4ce261 (0x72e bytes) 000293[1] 1788535219.841603 trunc from 0x828899 to 0x58d7a7 (0x29b0f1 bytes) 000294[0] 1788535219.877742 mapread 0x1194c0 thru 0x11b206 (0x1d47 bytes) 000296[0] 1788535219.959334 write 0x52d9f2 thru 0x52f4e9 (0x1af8 bytes) 000297[1] 1788535220.024457 trunc from 0x58d7a8 to 0x66f382 (0xe1bdb bytes) 000300[0] 1788535220.218105 mapread 0x29cb77 thru 0x29d7d3 (0xc5d bytes) 000301[1] 1788535220.292819 trunc from 0x66f383 to 0x025b47 (0x64983b bytes) 000303[0] 1788535220.389285 trunc from 0x025b48 to 0x21d741 (0x1f7bfa bytes) 000304[1] 1788535220.480134 read 0x03f9cf thru 0x0403a2 (0x9d4 bytes) 000306[1] 1788535220.592666 trunc from 0x21d742 to 0x629345 (0x40bc04 bytes) truncating to largest ever: 0x9bff21 000308[0] 1788535220.724233 trunc from 0x629346 to 0x9bff20 (0x396bdb bytes) 000310[1] 1788535220.792685 punch from 0x5506d5 to 0x5514c2 (0xdee bytes) 000311[0] 1788535220.849045 read 0x3f0453 thru 0x3f1c3f (0x17ed bytes) 000313[0] 1788535220.918041 mapread 0x85f26c thru 0x85f6e8 (0x47d bytes) 000314[1] 1788535220.966739 mapwrite 0x934788 thru 0x935cb1 (0x152a bytes) 000315[0] 1788535221.190305 mapread 0x7e0938 thru 0x7e157d (0xc46 bytes) 000316[0] 1788535221.249419 mapwrite 0x5f2933 thru 0x5f3db5 (0x1483 bytes) 000317[0] 1788535221.412466 punch from 0x49a238 to 0x49be00 (0x1bc9 bytes) 000318[0] 1788535221.446195 read 0x34a2bb thru 0x34a7cf (0x515 bytes) 000320[1] 1788535221.513032 write 0x37a40f thru 0x37bf6b (0x1b5d bytes) 000325[0] 1788535221.630352 mapread 0x500eb2 thru 0x501f03 (0x1052 bytes) 000327[0] 1788535221.693622 mapwrite 0x13b495 thru 0x13cfe4 (0x1b50 bytes) 000329[1] 1788535221.808374 trunc from 0x9bff21 to 0x473bf0 (0x54c330 bytes) 000331[1] 1788535221.955563 read 0x385a26 thru 0x387784 (0x1d5f bytes) 000332[1] 1788535222.022218 punch from 0x440504 to 0x440ce1 (0x7de bytes) 000335[0] 1788535222.163709 punch from 0x44b684 to 0x44cc98 (0x1615 bytes) 000336[1] 1788535222.246761 mapwrite 0x0dd919 thru 0x0df250 (0x1938 bytes) 000337[0] 1788535222.359784 mapwrite 0x002c99 thru 0x003374 (0x6dc bytes) 000338[1] 1788535222.460962 read 0x4144db thru 0x41551d (0x1043 bytes) 000340[0] 1788535222.545519 punch from 0x349c7b to 0x349e24 (0x1aa bytes) 000341[1] 1788535222.550065 punch from 0x009724 to 0x00b708 (0x1fe5 bytes) 000342[0] 1788535222.575004 punch from 0x232879 to 0x2347d9 (0x1f61 bytes) 000343[1] 1788535222.606338 punch from 0x00d259 to 0x00ef5c (0x1d04 bytes) 000344[1] 1788535222.644777 read 0x039b9d thru 0x039f6d (0x3d1 bytes) 000345[0] 1788535222.671431 mapwrite 0x1ac7bd thru 0x1adc83 (0x14c7 bytes) 000346[1] 1788535222.768298 trunc from 0x473bf1 to 0x5eb738 (0x177b48 bytes) 000347[1] 1788535222.839693 punch from 0x30d6fa to 0x30f626 (0x1f2d bytes) 000348[1] 1788535222.849320 punch from 0x263de3 to 0x264e55 (0x1073 bytes) 000351[1] 1788535222.896851 punch from 0x16941e to 0x16b128 (0x1d0b bytes) 000353[0] 1788535222.919198 read 0x446142 thru 0x447cc8 (0x1b87 bytes) 000354[0] 1788535222.961460 read 0x2a7c13 thru 0x2a837a (0x768 bytes) 000355[1] 1788535222.992266 mapwrite 0x8f8ccf thru 0x8f9482 (0x7b4 bytes) 000356[0] 1788535223.164064 trunc from 0x8f9483 to 0x52891e (0x3d0b64 bytes) 000357[1] 1788535223.224491 punch from 0x466804 to 0x467668 (0xe65 bytes) 000358[0] 1788535223.273121 trunc from 0x52891f to 0x1c3bb7 (0x364d67 bytes) 000359[0] 1788535223.304485 read 0x0d2ffd thru 0x0d311d (0x121 bytes) 000362[0] 1788535223.352671 read 0x046711 thru 0x0471ec (0xadc bytes) 000363[1] 1788535223.353605 write 0x197d91 thru 0x1980ff (0x36f bytes) 000365[1] 1788535223.431764 trunc from 0x1c3bb8 to 0x5ac286 (0x3e86cf bytes) 000366[1] 1788535223.496218 punch from 0x461a30 to 0x461fb3 (0x584 bytes) 000369[1] 1788535223.529091 punch from 0x421618 to 0x4235b9 (0x1fa2 bytes) 000372[0] 1788535223.568676 mapread 0x41c7cc thru 0x41d6ae (0xee3 bytes) 000373[0] 1788535223.610211 read 0x08c5c4 thru 0x08d097 (0xad4 bytes) 000374[1] 1788535223.641027 read 0x0b500a thru 0x0b53a2 (0x399 bytes) 000375[0] 1788535223.647762 read 0x4d26e6 thru 0x4d3b1d (0x1438 bytes) 000376[1] 1788535223.654189 mapwrite 0x66f631 thru 0x67042c (0xdfc bytes) 000378[1] 1788535223.964809 trunc from 0x67042d to 0x65848b (0x17fa1 bytes) 000380[0] 1788535224.094891 mapwrite 0x874a22 thru 0x875986 (0xf65 bytes) 000381[0] 1788535224.363721 read 0x5e5eb8 thru 0x5e7686 (0x17cf bytes) 000383[0] 1788535224.421537 mapwrite 0x8535ca thru 0x8555c6 (0x1ffd bytes) 000386[0] 1788535224.581219 mapread 0x711b0b thru 0x712a03 (0xef9 bytes) 000390[0] 1788535224.663085 read 0x2c4bf8 thru 0x2c561f (0xa28 bytes) 000394[1] 1788535224.697742 write 0x391413 thru 0x39258b (0x1179 bytes) 000395[0] 1788535224.739524 write 0x02f518 thru 0x031299 (0x1d82 bytes) 000396[1] 1788535224.779457 read 0x7f0cbf thru 0x7f27a9 (0x1aeb bytes) 000398[1] 1788535224.814118 trunc from 0x875987 to 0x585283 (0x2f0703 bytes) 000399[1] 1788535224.916810 mapwrite 0x4c51fc thru 0x4c704c (0x1e51 bytes) 000400[0] 1788535225.015688 punch from 0x3cbf1a to 0x3cd671 (0x1758 bytes) 000401[1] 1788535225.083067 trunc from 0x585284 to 0x3a690a (0x1de979 bytes) 000402[1] 1788535225.183640 punch from 0x1121f4 to 0x113b66 (0x1973 bytes) 000405[0] 1788535225.239685 punch from 0x1a75f0 to 0x1a8cf2 (0x1703 bytes) 000407[1] 1788535225.296217 punch from 0x28dd2b to 0x28f1c7 (0x149d bytes) 000408[1] 1788535225.365264 write 0x1937a8 thru 0x194814 (0x106d bytes) 000409[0] 1788535225.383428 write 0x52ffc2 thru 0x5307ed (0x82c bytes) 000410[0] 1788535225.428499 trunc from 0x5307ee to 0x159ba9 (0x3d6c44 bytes) 000414[0] 1788535225.554236 punch from 0x04c335 to 0x04c44d (0x119 bytes) 000415[0] 1788535225.580115 mapread 0x01f0c0 thru 0x020a33 (0x1974 bytes) 000416[0] 1788535225.594158 mapwrite 0x858a9f thru 0x859605 (0xb67 bytes) 000417[0] 1788535225.729693 read 0x0b2fcc thru 0x0b4d98 (0x1dcd bytes) 000422[0] 1788535225.867539 read 0x39c646 thru 0x39df66 (0x1921 bytes) 000423[0] 1788535225.918530 punch from 0x818655 to 0x819fc3 (0x196f bytes) 000426[1] 1788535225.985376 punch from 0x10487d to 0x105e3c (0x15c0 bytes) 000427[1] 1788535226.015192 punch from 0x03651d to 0x036986 (0x46a bytes) 000428[1] 1788535226.037075 trunc from 0x859606 to 0x14736e (0x712297 bytes) 000429[1] 1788535226.083340 write 0x8d1b8e thru 0x8d2d21 (0x1194 bytes) 000430[1] 1788535226.139691 punch from 0x5de8b4 to 0x5df2e8 (0xa35 bytes) 000432[1] 1788535226.169390 write 0x223055 thru 0x22365d (0x609 bytes) 000433[0] 1788535226.179164 mapread 0x30f55d thru 0x310de5 (0x1889 bytes) 000435[0] 1788535226.267669 punch from 0x146050 to 0x146100 (0xb1 bytes) 000436[1] 1788535226.341132 punch from 0x6c8768 to 0x6c8c80 (0x519 bytes) 000437[1] 1788535226.388742 write 0x58c457 thru 0x58cf22 (0xacc bytes) 000439[0] 1788535226.461778 punch from 0x3a1389 to 0x3a1da4 (0xa1c bytes) 000440[1] 1788535226.493216 mapwrite 0x8a3dd3 thru 0x8a54cc (0x16fa bytes) 000442[0] 1788535226.667197 write 0x8120cc thru 0x813d7c (0x1cb1 bytes) 000445[1] 1788535226.757285 punch from 0x2a47f2 to 0x2a614a (0x1959 bytes) 000446[1] 1788535226.801885 write 0x4a27b4 thru 0x4a3bde (0x142b bytes) 000450[0] 1788535226.871243 trunc from 0x8d2d22 to 0x1713f2 (0x76192f bytes) 000451[0] 1788535226.968459 mapwrite 0x486361 thru 0x488262 (0x1f02 bytes) 000452[0] 1788535227.062198 mapread 0x03b7c8 thru 0x03ce0c (0x1645 bytes) 000453[0] 1788535227.074617 mapwrite 0x498933 thru 0x49a5a8 (0x1c76 bytes) 000454[0] 1788535227.194716 mapwrite 0x695592 thru 0x697554 (0x1fc3 bytes) 000456[1] 1788535227.311785 trunc from 0x697555 to 0x6e305a (0x4bb06 bytes) 000459[0] 1788535227.400024 mapread 0x673ea9 thru 0x67541a (0x1572 bytes) 000463[1] 1788535227.538994 write 0x7c593d thru 0x7c60a9 (0x76d bytes) 000464[0] 1788535227.569875 trunc from 0x7c60aa to 0x77e760 (0x47949 bytes) 000465[1] 1788535227.622293 punch from 0x47b278 to 0x47c24d (0xfd6 bytes) 000469[0] 1788535227.742715 read 0x5a8ee5 thru 0x5aa18e (0x12aa bytes) 000470[1] 1788535227.763438 read 0x72107e thru 0x7224ef (0x1472 bytes) 000471[0] 1788535227.770502 mapread 0x73a6cc thru 0x73ac24 (0x559 bytes) 000472[0] 1788535227.790696 write 0x608bf0 thru 0x609b69 (0xf7a bytes) 000473[1] 1788535227.832169 mapwrite 0x05ba7d thru 0x05c1bc (0x740 bytes) 000477[1] 1788535228.004358 read 0x43cad9 thru 0x43d663 (0xb8b bytes) 000478[1] 1788535228.071946 punch from 0x61d0a9 to 0x61d560 (0x4b8 bytes) 000480[0] 1788535228.112264 mapwrite 0x085a53 thru 0x0875a2 (0x1b50 bytes) 000481[0] 1788535228.249270 mapread 0x3a453d thru 0x3a48c8 (0x38c bytes) 000484[1] 1788535228.389060 punch from 0x520b21 to 0x521e21 (0x1301 bytes) 000485[1] 1788535228.481156 read 0x3cfa3c thru 0x3d0cd1 (0x1296 bytes) 000486[0] 1788535228.640466 mapread 0x0cb24a thru 0x0cb862 (0x619 bytes) 000488[1] 1788535228.718750 punch from 0x5800bc to 0x581a65 (0x19aa bytes) 000489[1] 1788535228.723570 trunc from 0x77e761 to 0x2cb803 (0x4b2f5d bytes) 000490[0] 1788535228.802156 trunc from 0x2cb804 to 0x2f14cf (0x25ccc bytes) 000491[1] 1788535228.890295 mapread 0x27fe1b thru 0x280442 (0x628 bytes) 000492[1] 1788535228.939007 punch from 0x16588f to 0x16615b (0x8cd bytes) 000493[0] 1788535228.984688 write 0x11896d thru 0x11a1d1 (0x1865 bytes) 000495[0] 1788535229.012375 write 0x087fa9 thru 0x087ff8 (0x50 bytes) 000497[1] 1788535229.019289 trunc from 0x2f14d0 to 0x716c85 (0x4257b6 bytes) 000498[0] 1788535229.098649 read 0x388aff thru 0x38a222 (0x1724 bytes) 000499[1] 1788535229.131721 read 0x4ae7b9 thru 0x4aebdc (0x424 bytes) All operations completed A-OK! PASS 16b (36s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 11:20:35 (1788535235) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 11:20:40 (1788535240) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.607716 s, 13.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.492947 s, 17.0 MB/s ldlm.namespaces.MGC192.168.206.130@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9569856e6800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9569926b4800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9569926b4800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9569926b4800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.6134 s, 7.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.32493 s, 24.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.7511 s, 38.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.62694 s, 15.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.19806 s, 12.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.24863 s, 24.7 MB/s PASS 16d (55s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 11:21:35 (1788535295) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.430932 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128489 s, 8.2 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 11:21:43 (1788535303) TIMEOUT PASS 16f (30s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 11:22:13 (1788535333) TIMEOUT PASS 16g (27s) == sanityn test 16h: mmap read after truncate file ======= 11:22:40 (1788535360) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (8s) == sanityn test 16i: read after truncate file ============ 11:22:48 (1788535368) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 11:22:57 (1788535377) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.72422 s, 15.2 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.22174 s, 11.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.28266 s, 8.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.09647 s, 12.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.58546 s, 10.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.62024 s, 10.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.48525 s, 7.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.40497 s, 18.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.96544 s, 13.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.133 s, 6.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.22522 s, 6.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.29584 s, 11.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.55534 s, 10.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.85233 s, 9.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.95593 s, 8.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.74308 s, 9.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.94355 s, 8.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.37002 s, 7.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.40465 s, 7.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (43s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 11:23:40 (1788535420) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! + fsx 1 done: rc=0 Fri Sep 4 11:24:26 EDT 2026 All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 2 done: rc=0 Fri Sep 4 11:24:27 EDT 2026 + fsx 3 done: rc=0 Fri Sep 4 11:24:27 EDT 2026 + fsx 4 done: rc=0 Fri Sep 4 11:24:27 EDT 2026 + fsx 5 done: rc=0 Fri Sep 4 11:24:27 EDT 2026 PASS 16k (56s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 11:24:37 (1788535477) fail_loc=0x8000030a PASS 17 (10s) == sanityn test 18: mmap sanity check =========================================================================================== 11:24:47 (1788535487) mmap test1: basic mmap operation (PASS, 0.17312s) mmap test2: MAP_PRIVATE not write back (PASS, 0.11227s) mmap test3: concurrent mmap ops on two nodes (PASS, 10.038s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.5803s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.2065s) mmap test6: check mmap write/read content on two nodes (PASS, 0.30146s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.6744s) mmap test11: multi-thread shared mmap (PASS, 1.4722s) mmap test10: mtime not change for readonly mmap access (PASS, 1.154s) PASS 18 (28s) == sanityn test 19: test concurrent uncached read races ========================================================================= 11:25:15 (1788535515) oleg630-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg630-client: oleg630-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 11:25:19 (1788535519) PASS 20 (7s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 11:25:26 (1788535526) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (6s) == sanityn test 23: others should see updated atime while another read============================================================== 11:25:32 (1788535532) now is 1788535535 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 new atime is 1788535596 PASS 23 (68s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 11:26:40 (1788535600) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (6s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 11:26:46 (1788535606) PASS 24b (7s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 11:26:53 (1788535613) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (6s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 11:27:00 (1788535620) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 11:27:02 (1788535622) utime: good mknod times 1788535623 <= 1788535624 <= 1788535624 for /mnt/lustre/f26a utime: good mknod times 1788535623 <= 1788535624 <= 1788535624 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (8s) == sanityn test 26b: sync mtime between ost and mds ====== 11:27:10 (1788535630) PASS 26b (9s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 11:27:19 (1788535639) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-04 11:27:20.000000000 -0400 Birth: 2026-09-04 11:27:19.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-04 11:27:20.000000000 -0400 Birth: 2026-09-04 11:27:19.000000000 -0400 PASS 26c (7s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 11:27:28 (1788535648) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 11:27:37 (1788535657) fail_loc=0x314 PASS 31a (9s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 11:27:46 (1788535666) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg630-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.206.130@tcp:/lustre 7542784 39936 7494656 1% /mnt/lustre2 PASS 31b (25s) == sanityn test 31r: open-rename(replace) race =========== 11:28:11 (1788535691) -rw-r--r-- 1 root root 0 Sep 4 11:28 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (12s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 11:28:23 (1788535703) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43150 Terminated tail -f $tfile PASS 31s (9s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 11:28:32 (1788535712) PASS 31t (9s) == sanityn test 32b: lockless i/o ======================== 11:28:41 (1788535721) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (5s) == sanityn test 32c: contention detection ================ 11:28:46 (1788535726) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.75517 s, 29.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.80754 s, 29.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.93618 s, 17.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.15901 s, 16.6 MB/s contention_seconds 4, contention_events 22 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.34329 s, 22.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.67086 s, 19.6 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.49815 s, 35.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.52634 s, 34.3 MB/s PASS 32c (35s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 11:29:23 (1788535763) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 11:29:27 (1788535767) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 11:29:31 (1788535771) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 11:29:35 (1788535775) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (4s) == sanityn test 34: no lock timeout under IO ============= 11:29:39 (1788535779) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg630-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9569856e6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9569856e6800.ost_server_uuid in IDLE state after 0 sec oleg630-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9569856e6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9569856e6800.ost_server_uuid in IDLE state after 0 sec oleg630-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9569856e6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9569856e6800.ost_server_uuid in IDLE state after 0 sec oleg630-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9569856e6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9569856e6800.ost_server_uuid in IDLE state after 0 sec oleg630-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9569856e6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9569856e6800.ost_server_uuid in IDLE state after 0 sec oleg630-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9569856e6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9569856e6800.ost_server_uuid in IDLE state after 0 sec PASS 34 (141s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 11:32:00 (1788535920) fail_loc=0x80000317 Race attempt 0 Wait for 50463 50476 for 60 sec... - open/close 727 (time 1788535938.36 total 10.00 last 72.67) - open/close 1428 (time 1788535948.37 total 20.01 last 70.08) - open/close 2162 (time 1788535958.37 total 30.02 last 73.34) - open/close 2847 (time 1788535968.39 total 40.03 last 68.43) - open/close 3448 (time 1788535978.39 total 50.03 last 60.06) total: 4000 open/close in 59.49 seconds: 67.24 ops/second /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1855: 50463 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.206.130@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.206.130@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 PASS 35 (73s) == sanityn test 36: handle ESTALE/open-unlink correctly == 11:33:13 (1788535993) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg630-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.63283 s, 32.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.25314 s, 41.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7494656) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.73067 s, 30.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7490560) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.74018 s, 30.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7494656) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.4225 s, 36.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7489536) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.50634 s, 34.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.77709 s, 29.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7496704) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.49394 s, 35.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7490560) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.49441 s, 35.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(8) *** before(7492608) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.48932 s, 35.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7489536) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.21729 s, 43.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(10) *** before(7490560) after_dd(7441408) after(7492608) PASS 36 (302s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 11:38:15 (1788536295) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 - create 1366 (time 1788536309.40 total 10.00 last 136.59) - create 2665 (time 1788536319.40 total 20.01 last 129.80) - create 4019 (time 1788536329.40 total 30.01 last 135.38) - create 5284 (time 1788536339.41 total 40.01 last 126.45) - create 6590 (time 1788536349.41 total 50.02 last 130.53) - create 7730 (time 1788536359.41 total 60.02 last 113.99) - create 8825 (time 1788536369.42 total 70.03 last 109.40) - create 9744 (time 1788536379.43 total 80.03 last 91.86) total: 10000 create in 82.08 seconds: 121.83 ops/second PASS 37 (95s) == sanityn test 39a: file mtime does not change after rename ========================================================== 11:39:51 (1788536391) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 11:39:59 (1788536399) repeat after cancel_lru_locks PASS 39b (10s) == sanityn test 39c: check truncate mtime update ================================================================================ 11:40:09 (1788536409) repeat after cancel_lru_locks PASS 39c (12s) == sanityn test 39d: sync write should update mtime ====== 11:40:21 (1788536421) fail_loc=0x411 fail_loc=0 PASS 39d (8s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 11:40:31 (1788536431) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (18s) == sanityn test 40c: pdirops: link and others ======================================================================== 11:40:50 (1788536450) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (18s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 11:41:08 (1788536468) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (15s) == sanityn test 40e: pdirops: rename and others ======================================================================== 11:41:23 (1788536483) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (16s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 11:41:39 (1788536499) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (13s) == sanityn test 41b: pdirops: create vs create ======================================================================== 11:41:52 (1788536512) open: File exists Conflict PASS 41b (12s) == sanityn test 41c: pdirops: create vs link ======================================================================== 11:42:05 (1788536525) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (12s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 11:42:17 (1788536537) Conflict PASS 41d (15s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 11:42:32 (1788536552) Conflict PASS 41e (12s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 11:42:44 (1788536564) Conflict PASS 41f (13s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 11:42:57 (1788536577) Conflict PASS 41g (12s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 11:43:09 (1788536589) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (13s) == sanityn test 41i: reint_open: create vs create ======== 11:43:22 (1788536602) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1099s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 12:01:41 (1788537701) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (14s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 12:01:55 (1788537715) open: File exists Conflict PASS 42b (13s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 12:02:08 (1788537728) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (14s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 12:02:22 (1788537742) Conflict PASS 42d (13s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 12:02:36 (1788537756) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (17s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 12:02:52 (1788537772) Conflict PASS 42f (14s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 12:03:06 (1788537786) Conflict PASS 42g (12s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 12:03:19 (1788537799) Conflict PASS 42h (14s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 12:03:33 (1788537813) PASS 43a (75s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 12:04:48 (1788537888) Conflict PASS 43b (15s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 12:05:03 (1788537903) Conflict PASS 43c (14s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 12:05:17 (1788537917) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (16s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 12:05:33 (1788537933) Conflict PASS 43e (18s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 12:05:51 (1788537951) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (15s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 12:06:06 (1788537966) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (13s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 12:06:20 (1788537980) Conflict PASS 43h (14s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 12:06:34 (1788537994) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 12:06:38 (1788537998) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (144s) == sanityn test 43k: unlink vs create ==================== 12:09:02 (1788538142) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1163s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 12:28:25 (1788539305) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Sep 4 12:28:33 EDT 2026 PASS 44a (13s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 12:28:38 (1788539318) open: File exists Conflict PASS 44b (13s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 12:28:51 (1788539331) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (14s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 12:29:05 (1788539345) Conflict PASS 44d (12s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 12:29:17 (1788539357) Conflict PASS 44e (14s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 12:29:31 (1788539371) Conflict PASS 44f (13s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 12:29:44 (1788539384) Conflict PASS 44g (12s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 12:29:56 (1788539396) Conflict PASS 44h (13s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 12:30:09 (1788539409) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (3s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 12:30:12 (1788539412) PASS 45a (126s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 12:32:18 (1788539538) Conflict PASS 45b (14s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 12:32:32 (1788539552) Conflict PASS 45c (13s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 12:32:45 (1788539565) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (13s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 12:32:59 (1788539579) Conflict PASS 45e (13s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 12:33:12 (1788539592) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (13s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 12:33:25 (1788539605) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (13s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 12:33:38 (1788539618) Conflict PASS 45h (12s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 12:33:51 (1788539631) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 12:33:54 (1788539634) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1164s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 12:53:18 (1788540798) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (13s) == sanityn test 46b: pdirops: link vs create ======================================================================== 12:53:31 (1788540811) open: File exists Conflict PASS 46b (12s) == sanityn test 46c: pdirops: link vs link ======================================================================== 12:53:43 (1788540823) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (14s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 12:53:57 (1788540837) Conflict PASS 46d (13s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 12:54:10 (1788540850) Conflict PASS 46e (14s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 12:54:24 (1788540864) Conflict PASS 46f (15s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 12:54:39 (1788540879) Conflict PASS 46g (14s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 12:54:53 (1788540893) Conflict PASS 46h (14s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 12:55:07 (1788540907) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (4s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 12:55:11 (1788540911) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (3s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 12:55:14 (1788540914) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 12:55:17 (1788540917) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (4s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 12:55:21 (1788540921) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 12:55:24 (1788540924) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 12:55:27 (1788540927) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (4s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 12:55:31 (1788540931) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 12:55:35 (1788540935) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0994792 s, 103 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (12s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 12:55:47 (1788540947) 0+1 records in 0+1 records out 158 bytes copied, 0.0358715 s, 4.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218332) - No such process PASS 51a (9s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 12:55:56 (1788540956) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.010748 s, 95.3 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690026 s, 15.2 MB/s 3145728 PASS 51b (20s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 12:56:16 (1788540976) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0196733 s, 52.1 kB/s PASS 51c (11s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 12:56:28 (1788540988) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466479 s, 22.5 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (7s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 12:56:35 (1788540995) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2451 0x993 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2451 0x993 0x240000400 read lease(1) has applied. PASS 51e (9s) == sanityn test 54: rename locking ======================= 12:56:44 (1788541004) llite.lustre-ffff9569856e6800.xattr_cache=1 llite.lustre-ffff9569926b4800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 12:56:46.000000000 -0400 Modify: 2026-09-04 12:56:46.000000000 -0400 Change: 2026-09-04 12:56:46.000000000 -0400 Birth: 2026-09-04 12:56:45.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 12:56:52.000000000 -0400 Modify: 2026-09-04 12:56:54.000000000 -0400 Change: 2026-09-04 12:56:54.000000000 -0400 Birth: 2026-09-04 12:56:51.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 12:56:59.000000000 -0400 Modify: 2026-09-04 12:56:59.000000000 -0400 Change: 2026-09-04 12:56:59.000000000 -0400 Birth: 2026-09-04 12:56:58.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 12:56:59.000000000 -0400 Modify: 2026-09-04 12:57:00.000000000 -0400 Change: 2026-09-04 12:57:00.000000000 -0400 Birth: 2026-09-04 12:56:58.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 12:57:05.000000000 -0400 Modify: 2026-09-04 12:57:06.000000000 -0400 Change: 2026-09-04 12:57:06.000000000 -0400 Birth: 2026-09-04 12:57:04.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 12:57:05.000000000 -0400 Modify: 2026-09-04 12:57:06.000000000 -0400 Change: 2026-09-04 12:57:06.000000000 -0400 Birth: 2026-09-04 12:57:04.000000000 -0400 PASS 54 (33s) == sanityn test 55a: rename vs unlink target dir ========= 12:57:18 (1788541038) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (14s) == sanityn test 55b: rename vs unlink source dir ========= 12:57:31 (1788541051) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (12s) == sanityn test 55c: rename vs unlink orphan target dir == 12:57:43 (1788541063) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (20s) == sanityn test 55d: rename file vs link ================= 12:58:03 (1788541083) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (16s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 12:58:19 (1788541099) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (4s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 12:58:23 (1788541103) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (19s) == sanityn test 55g: rename: race with trylock =========== 12:58:42 (1788541122) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (22s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 12:59:04 (1788541144) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788541146 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.565177 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (117s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 13:01:02 (1788541262) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788541270 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.03453 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (136s) == sanityn test 60: Verify data_version behaviour ======== 13:03:17 (1788541397) PASS 60 (12s) == sanityn test 70a: cd directory && rm directory ======== 13:03:30 (1788541410) PASS 70a (10s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 13:03:40 (1788541420) PASS 70b (9s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 13:03:49 (1788541429) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0198471 s, 3.3 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 13:03:53 (1788541433) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.023487 s, 2.8 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 13:03:56 (1788541436) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (4s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 13:04:00 (1788541440) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0259383 s, 2.5 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (3s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 13:04:03 (1788541443) llite.lustre-ffff9569856e6800.xattr_cache=1 llite.lustre-ffff9569926b4800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (8s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 13:04:11 (1788541451) llite.lustre-ffff9569856e6800.xattr_cache=1 llite.lustre-ffff9569926b4800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (6s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 13:04:18 (1788541458) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0160227 s, 63.9 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0183982 s, 55.7 kB/s 231157: get lock2 231158: get lock1 231157: done 231157: get lock1 231158: done 231158: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231158: cannot get lock2: Resource deadlock avoided 231158: exit rc=0 231157: done 231157: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 13:04:28 (1788541468) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0878196 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555039 s, 18.9 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 231738 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (12s) == sanityn test 76: Verify MDT open_files listing ======== 13:04:40 (1788541480) opening files: ........ closing files: ........ PASS 76 (80s) == sanityn test 77a: check FIFO NRS policy =============== 13:06:00 (1788541560) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.508138 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783546 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718448 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187354 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2128 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237836 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221684 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193991 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270324 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304346 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283461 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340376 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332519 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35367 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319771 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379564 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352375 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259185 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331011 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951703 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459301 s, 22.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111162 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571262 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591327 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621709 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687361 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11281 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895487 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894894 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096749 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098935 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980941 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114466 s, 9.2 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (9s) == sanityn test 77b: check CRR-N NRS policy ============== 13:06:10 (1788541570) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.476282 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552158 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582961 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09527 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174845 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255166 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180291 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197454 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304729 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281021 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317295 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348081 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348869 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316472 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277365 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392753 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102779 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294461 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134777 s, 77.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419528 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465059 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150428 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426437 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050318 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.473146 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953526 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610227 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189368 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567386 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701064 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778219 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764663 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085047 s, 12.3 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg630-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg630-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.530652 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975495 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677021 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825819 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164241 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867558 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540362 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156548 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188565 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137882 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244703 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25555 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224463 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16123 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251302 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33287 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283361 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052006 s, 20.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407353 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124037 s, 84.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934029 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974645 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501384 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569533 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110275 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551864 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811455 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089564 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010804 s, 97.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108137 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912146 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560974 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153444 s, 6.8 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg630-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg630-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (14s) == sanityn test 77c: check ORR NRS policy ================ 13:06:23 (1788541583) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.473784 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877402 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100654 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733932 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127168 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157728 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129911 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217456 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183264 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265649 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21223 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220309 s, 4.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242348 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243231 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246372 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265478 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240868 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882042 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316494 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143222 s, 73.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150067 s, 69.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452922 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040835 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655012 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170019 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00629972 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947456 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686883 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594867 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130398 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00808517 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12008 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133 s, 7.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.598644 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522583 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108205 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114559 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118765 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123723 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976787 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150013 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125456 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146914 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164797 s, 6.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364389 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356475 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307161 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.423724 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00748484 s, 140 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837234 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407141 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00839326 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936274 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.441251 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024061 s, 43.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12281 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164696 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04193 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124922 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264967 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00971147 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00738478 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14193 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121677 s, 86.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00899361 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00459515 s, 228 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg630-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg630-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.565972 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139453 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151348 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101211 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703011 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177856 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135098 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662532 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111811 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245569 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12715 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215474 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288358 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157442 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21044 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39159 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281652 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698589 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699833 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510969 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528682 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356404 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419913 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140113 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509707 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054005 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739107 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625795 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643105 s, 16.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070873 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770104 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766792 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813488 s, 12.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (17s) == sanityn test 77d: check TRR nrs policy ================ 13:06:40 (1788541600) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.716771 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840577 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121252 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108518 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195881 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205041 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210352 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18755 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305205 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284685 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312546 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306092 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291163 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375329 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273523 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247297 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30949 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746449 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912763 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364721 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10768 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583826 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443664 s, 23.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061912 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437234 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118687 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585814 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759496 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611437 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612238 s, 17.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580801 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738424 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747988 s, 14.0 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg630-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg630-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.507273 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905728 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923898 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138063 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181437 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147516 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137151 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191217 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11582 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194468 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151135 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164904 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246306 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24851 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179625 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247191 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282512 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103646 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123766 s, 84.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861717 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703486 s, 14.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118326 s, 88.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139602 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763199 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860568 s, 12.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146389 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13242 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777377 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318902 s, 32.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082357 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975525 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587267 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797134 s, 13.2 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.572579 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944647 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543134 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123531 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186012 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211411 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307964 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210305 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213256 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325673 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416211 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371884 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287636 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429414 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.436714 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360687 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.466178 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125481 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163887 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150915 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176098 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171385 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157701 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137782 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150207 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147765 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141992 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15324 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155857 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193363 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20168 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15557 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178593 s, 5.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg630-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg630-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (18s) == sanityn test 77e: check TBF NID nrs policy ============ 13:06:58 (1788541618) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.206.30@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.440294 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644616 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456241 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154263 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151822 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137292 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158685 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148346 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147169 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192864 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181118 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196057 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238664 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178641 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189208 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216716 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31896 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409614 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462631 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804613 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551498 s, 19.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436287 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829103 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069736 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106912 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719464 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814662 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11952 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779642 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756166 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112562 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117585 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887011 s, 11.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.519253 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703936 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884427 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153532 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183795 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075175 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108916 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158566 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21844 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201773 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271303 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323771 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363302 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31234 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339625 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303305 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412608 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581666 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280606 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559391 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102532 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456732 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612998 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034711 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626924 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408976 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545717 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516769 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740886 s, 14.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639293 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918886 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643807 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736837 s, 14.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.510456 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100692 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127567 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189306 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173796 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202764 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249213 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21075 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310053 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246582 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298686 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31416 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345833 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3754 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277958 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363042 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348649 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073043 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520304 s, 20.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058466 s, 17.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980823 s, 10.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04525 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693056 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101549 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520603 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491929 s, 21.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383062 s, 27.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353746 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602559 s, 17.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502064 s, 20.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474767 s, 22.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719968 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10414 s, 10.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=3 removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.539458 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809131 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106698 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104337 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160356 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177449 s, 5.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211855 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241172 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262439 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291892 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271037 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252781 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298491 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366269 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375538 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293981 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.424157 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796273 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773818 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036142 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659427 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070414 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809966 s, 12.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880396 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852432 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151448 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924625 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100827 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873649 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875641 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960619 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109368 s, 95.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141292 s, 7.4 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg630-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg630-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (23s) == sanityn test 77f: check TBF JobID nrs policy ========== 13:07:21 (1788541641) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg630-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.402574 s, 41.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858964 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104626 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18996 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175764 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222941 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138104 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226231 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206815 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319222 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265259 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3789 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359082 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30605 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.413114 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374117 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.516538 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128149 s, 8.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00240213 s, 437 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00920528 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184572 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00337418 s, 311 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109684 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198198 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872428 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109517 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757952 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754224 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070425 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629495 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917341 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596467 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097334 s, 10.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg630-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg630-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg630-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.551891 s, 30.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882861 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185134 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993448 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886159 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164086 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206501 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239426 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680732 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435771 s, 24.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410161 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101027 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886789 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849475 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129051 s, 8.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556144 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566482 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129313 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100937 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432268 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445367 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935744 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00639564 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665115 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581521 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00581518 s, 180 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.493664 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.527585 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.47034 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299119 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475155 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.514595 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41143 s, 2.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg630-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg630-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg630-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.527725 s, 31.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934095 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088342 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120968 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24712 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230646 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241916 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246665 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235924 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269672 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366635 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374382 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387632 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320577 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.427303 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578429 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072624 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134282 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.484764 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955927 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176006 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181952 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177052 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100433 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113279 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10255 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196345 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111315 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127032 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252751 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00733332 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103073 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.509802 s, 2.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg630-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.5105 s, 32.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695957 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967913 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105332 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156728 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860405 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227806 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212605 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169853 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258804 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227403 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232227 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260278 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248036 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326116 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331996 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374379 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=6 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124591 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125166 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906289 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467553 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422494 s, 24.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141324 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154071 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613608 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935966 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550137 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631828 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109994 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664557 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17667 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083027 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103632 s, 10.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg630-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg630-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (26s) == sanityn test 77g: Change TBF type directly ============ 13:07:47 (1788541667) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (13s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 13:08:00 (1788541680) oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg630-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg630-client: oleg630-server: ssh exited with exit code 2 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg630-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg630-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg630-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 PASS 77h (12s) == sanityn test 77i: Change rank of TBF rule ============= 13:08:13 (1788541693) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (25s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 13:08:38 (1788541718) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.948217 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116098 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131806 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141864 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218404 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280417 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282685 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261914 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260975 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282643 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342001 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340266 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269024 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334397 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297662 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338317 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.427958 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336123 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352339 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293886 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30716 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344425 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370663 s, 2.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324339 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327565 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37847 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332011 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342399 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34917 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347221 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355785 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349621 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357302 s, 2.9 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.30755 s, 12.6 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6399 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (67s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 13:09:45 (1788541785) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg630-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg630-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg630-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg630-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg630-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.33256 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704314 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750241 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108875 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11288 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13338 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147039 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14285 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126613 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176586 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127394 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261402 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260009 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152144 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23874 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253775 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291563 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089822 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130698 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792816 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143111 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141315 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102646 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935926 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966355 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935954 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150556 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929537 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150345 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984165 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11147 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112102 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141555 s, 7.4 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg630-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg630-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9428 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9559 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg630-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg630-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg630-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg630-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg630-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28659 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904087 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944433 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967148 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137594 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136902 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126601 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158924 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176146 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214528 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194001 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15946 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252164 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205974 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270815 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18518 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193727 s, 5.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090899 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815141 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911669 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101278 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111975 s, 9.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103968 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110324 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106711 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951783 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122103 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12245 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118638 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123036 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126344 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143966 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158643 s, 6.6 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg630-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg630-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9446 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9743 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (153s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 13:12:19 (1788541939) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg630-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg630-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg630-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg630-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg630-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27837 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130285 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136684 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12926 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140143 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139473 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162495 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169602 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199516 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224141 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205594 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221689 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239187 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241034 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245998 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281501 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307025 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979285 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103016 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103522 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111855 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100183 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804139 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1218 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13276 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105417 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11405 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104787 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146887 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149891 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149236 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168048 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148176 s, 7.1 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg630-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg630-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9613 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9649 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg630-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg630-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg630-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg630-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg630-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29398 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115241 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167589 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172081 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166145 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146385 s, 7.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302983 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277595 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201356 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21279 s, 4.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209452 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179665 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127792 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281351 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288057 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605288 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.510411 s, 2.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895524 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824988 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856083 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891852 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645212 s, 16.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712917 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757913 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110945 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928496 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945442 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115254 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123624 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133683 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133645 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104769 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108396 s, 9.7 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1663 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8368 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (156s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 13:14:54 (1788542094) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg630-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.8849 s, 19.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698218 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108177 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248657 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227796 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265949 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25279 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314049 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24684 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235955 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275601 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229266 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306152 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300664 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3893 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346648 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.500376 s, 2.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766732 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629164 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715004 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951587 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707673 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930802 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094819 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103792 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107971 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537005 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06175 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513591 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115424 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629583 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729248 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777768 s, 13.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.9291 s, 11.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0747 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.206.30@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.206.30@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.728419 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578568 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550411 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693812 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155375 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123838 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242949 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237299 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280589 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311033 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33345 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369127 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307423 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318626 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354033 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417793 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.474216 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209113 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196299 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194513 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202886 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204914 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209005 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217484 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204078 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216119 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162056 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201658 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201682 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235756 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22345 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210697 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238764 s, 4.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.48713 s, 11.1 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1102 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.206.30@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg630-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.641304 s, 26.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743098 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933844 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883832 s, 11.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420856 s, 24.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838107 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609827 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388652 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148515 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210009 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207146 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220382 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240592 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1482 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232798 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20126 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319208 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581777 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793807 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642877 s, 16.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645346 s, 16.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441294 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461909 s, 22.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057769 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103153 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613627 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532075 s, 19.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687551 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887298 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743781 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684323 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732495 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748538 s, 14.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg630-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg630-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.92365 s, 11.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.4156 s, 19.4 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg630-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.759064 s, 22.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668742 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215153 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370231 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158009 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447618 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306061 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728946 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669386 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111866 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228829 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301973 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349561 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412313 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.414834 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.395708 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345819 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877084 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878371 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710939 s, 14.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561009 s, 18.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517153 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560824 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100504 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102968 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463459 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990155 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552462 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686509 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00617504 s, 170 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678687 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133395 s, 78.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818403 s, 12.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.8697 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99919 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.79707 s, 11.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94346 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg630-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.06225 s, 15.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503575 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832853 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880743 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572046 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148712 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106009 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111562 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128663 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176726 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15421 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229944 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152279 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157153 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174133 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251733 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303299 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175197 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509163 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194052 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479199 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557337 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533931 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624986 s, 16.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667689 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111615 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709569 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903754 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00664728 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649972 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676083 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797758 s, 13.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749017 s, 14.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8887 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6446 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg630-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.783455 s, 21.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085335 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153471 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160493 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1117 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114451 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176549 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183737 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145164 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207442 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204722 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186779 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28312 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165329 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201019 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232399 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289804 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392097 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372148 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700027 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443436 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424544 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080457 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611508 s, 17.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520982 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545143 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574721 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419089 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631001 s, 16.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566972 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784837 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727539 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849644 s, 12.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.3991 s, 12.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9289 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg630-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.860893 s, 19.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103795 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110665 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170584 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151567 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177181 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150147 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061373 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167814 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176674 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164386 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107308 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226914 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254903 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25123 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235229 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222464 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337713 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425845 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048159 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089605 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545084 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767883 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896839 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136399 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707827 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959134 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102471 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102843 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145113 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102978 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163283 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186676 s, 5.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg630-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg630-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.3394 s, 9.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.3966 s, 9.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0432 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (442s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 13:22:17 (1788542537) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg630-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.760519 s, 22.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393974 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253033 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455003 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406378 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372451 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555076 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189167 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217119 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22162 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26441 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27725 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268907 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268331 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307475 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315696 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307081 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462979 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144834 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106895 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552085 s, 19.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107519 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104973 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128797 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134489 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108769 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101872 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263423 s, 39.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167697 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141529 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195069 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174889 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00957635 s, 109 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.56118 s, 11.0 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.023 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (55s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 13:23:12 (1788542592) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.206.30@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.206.30@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.866737 s, 19.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550948 s, 19.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754007 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105221 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147261 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168237 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144187 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220599 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212761 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258074 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213196 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260937 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214753 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219414 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19031 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246515 s, 4.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193523 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151438 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159755 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185296 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169044 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175288 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165671 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192344 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173056 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186663 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172948 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229664 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196549 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228561 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224852 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221717 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261618 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=3 removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg630-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg630-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.61226 s, 12.2 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93734 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (52s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 13:24:04 (1788542644) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.206.30@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg630-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.52501 s, 32.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664607 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871669 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641522 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109023 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988698 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102547 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910311 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999193 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129897 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188321 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212762 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218926 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231137 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24425 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234102 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27776 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111764 s, 93.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02574 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154703 s, 67.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174249 s, 60.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388713 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406615 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023888 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056261 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247045 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00446956 s, 235 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00492904 s, 213 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00232755 s, 451 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00779109 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00633871 s, 165 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00708854 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148911 s, 70.4 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg630-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg630-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.93847 s, 13.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.06157 s, 20.7 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (41s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 13:24:45 (1788542685) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg630-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.839647 s, 20.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950536 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955971 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554095 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130088 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138451 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201177 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104534 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211125 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23275 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308128 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325778 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238435 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316061 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322257 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404347 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21832 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693138 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664221 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786316 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080112 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862235 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110508 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865114 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902691 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685619 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738675 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831557 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862921 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850397 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870512 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842422 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922831 s, 11.4 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4814 s, 10.0 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99215 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.36582 s, 12.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8895 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (96s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 13:26:21 (1788542781) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg630-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.09861 s, 15.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134173 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141137 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116336 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171604 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184136 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867213 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967106 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293918 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214421 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294119 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106686 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241561 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229014 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306553 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366531 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242654 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334186 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240519 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106014 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461515 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0059357 s, 177 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136384 s, 76.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00713368 s, 147 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402653 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00596073 s, 176 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784941 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00410685 s, 255 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815244 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00556006 s, 189 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00728606 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101667 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119285 s, 87.9 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg630-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg630-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8663 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9887 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (78s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 13:27:39 (1788542859) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg630-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.822631 s, 20.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667013 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682541 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11532 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643286 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228319 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263509 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145124 s, 7.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288055 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171936 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121642 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216066 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400688 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41814 s, 2.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383858 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269158 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374198 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9569856e6800.lock_unused_count=6 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111526 s, 94.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168138 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035475 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170609 s, 61.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932578 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723692 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107923 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118934 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118978 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916291 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703066 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164566 s, 6.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9569856e6800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118425 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133769 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10888 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124798 s, 8.4 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg630-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg630-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.51709 s, 13.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89767 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg630-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.685869 s, 24.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891638 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701729 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17791 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221601 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201996 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291166 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259518 s, 4.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303032 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264454 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323725 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320542 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363935 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29709 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336642 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386679 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38237 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630278 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306872 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10902 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790057 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572511 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061594 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640613 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545729 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559065 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688503 s, 15.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795769 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818012 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866903 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906008 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958783 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118859 s, 8.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.9488 s, 8.8 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92771 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.3423 s, 9.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96941 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (144s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 13:30:03 (1788543003) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg630-client.virtnet /mnt/lustre (opts:) Stopping client oleg630-client.virtnet /mnt/lustre opts: Stopping clients: oleg630-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg630-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg630-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg630-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg630-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg630-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.30,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg630-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.30,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg630-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.30,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69504/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.30,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg630-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg630-server' oleg630-server: oleg630-server.virtnet: executing load_modules_local oleg630-server: kmemleak disabled oleg630-server: Loading modules from /home/green/git/lustre-release/lustre oleg630-server: detected 4 online CPUs by sysfs oleg630-server: MODOPTS_LIBCFS= oleg630-server: Force libcfs to create 2 CPU partitions oleg630-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg630-server: oleg630-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg630-server: oleg630-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg630-server: oleg630-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg630-client.virtnet: -o user_xattr,flock 192.168.206.130@tcp:/lustre /mnt/lustre Starting client oleg630-client.virtnet: -o user_xattr,flock 192.168.206.130@tcp:/lustre /mnt/lustre Started clients oleg630-client.virtnet: 192.168.206.130@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg630-client.virtnet: -o user_xattr,flock 192.168.206.130@tcp:/lustre /mnt/lustre2 Starting client oleg630-client.virtnet: -o user_xattr,flock 192.168.206.130@tcp:/lustre /mnt/lustre2 Started clients oleg630-client.virtnet: 192.168.206.130@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff956992743000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9569a0e80000.idle_timeout=debug osc.lustre-OST0001-osc-ffff956992743000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9569a0e80000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.490392 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790325 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11337 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128941 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1373 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127091 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672238 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107304 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148212 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204391 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180418 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222795 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266063 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217532 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161818 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179752 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237023 s, 4.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff956992743000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125763 s, 83.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300948 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452772 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462396 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00850652 s, 123 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525121 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360048 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029373 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490019 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437613 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675717 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692073 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915888 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00508677 s, 206 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00350591 s, 299 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00750494 s, 140 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.13637 s, 14.7 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.18008 s, 20.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.520566 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750905 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791353 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190708 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178189 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205302 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194776 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209631 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235254 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264284 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258579 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24206 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232843 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22192 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248084 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276818 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253654 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff956992743000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff956992743000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328071 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155224 s, 67.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530296 s, 19.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00464947 s, 226 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311434 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645812 s, 16.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281517 s, 37.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401081 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039038 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509118 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217311 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00472393 s, 222 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555727 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105343 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173696 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787971 s, 13.3 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.17894 s, 12.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92807 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (218s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 13:33:41 (1788543221) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg630-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (17s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 13:33:59 (1788543239) On MGS 192.168.206.130, active = nodemap.active=1 On MGS 192.168.206.130, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.455192 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899356 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963593 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895171 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107833 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859028 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175214 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253627 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241703 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277331 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294107 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2174 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311697 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314068 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256443 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323817 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242749 s, 4.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff956992743000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239117 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239295 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200139 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205677 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214093 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231773 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277889 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226196 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226716 s, 4.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226405 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231269 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233144 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234732 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27021 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257514 s, 4.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276631 s, 3.8 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.84966 s, 13.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.05099 s, 20.8 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.206.130, TBF.id = On MGS 192.168.206.130, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.555926 s, 30.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524459 s, 20.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497423 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706473 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825082 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581167 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116964 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287678 s, 36.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427631 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101218 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125102 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253575 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188456 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231195 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213636 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279413 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3266 s, 3.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff956992743000.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418469 s, 25.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422481 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433613 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457681 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746261 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966307 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684233 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424684 s, 24.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547664 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675897 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689154 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901324 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722943 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080818 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772726 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872548 s, 12.0 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.6184 s, 13.8 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.71069 s, 38.7 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg630-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg630-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 On MGS 192.168.206.130, active = nodemap.active=1 On MGS 192.168.206.130, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.676794 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730529 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111467 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795114 s, 13.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587681 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185555 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114554 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127192 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19597 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111762 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129765 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254076 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2453 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197594 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296012 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280658 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff956992743000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff956992743000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249659 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14094 s, 7.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163157 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1017 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118177 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116088 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106368 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159814 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128687 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178828 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134324 s, 7.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134506 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176021 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139391 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139307 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153311 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152814 s, 6.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff956992743000.lock_unused_count=5 removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg630-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg630-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.85463 s, 13.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91509 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.206.130, TBF.id = On MGS 192.168.206.130, active = nodemap.active=0 PASS 77kj (166s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 13:36:45 (1788543405) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (9s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 13:36:54 (1788543414) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg630-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.44323 s, 305 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg630-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0624 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (55s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 13:37:49 (1788543469) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.46122 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712793 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692002 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115142 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207921 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237932 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182941 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261667 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258606 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174233 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260322 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230764 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31665 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339995 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353058 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372415 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff956992743000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff956992743000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.450991 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462701 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310872 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330101 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713217 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344476 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370193 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756433 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423696 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434529 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441703 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463569 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854373 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519012 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158005 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646626 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145634 s, 7.2 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg630-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg630-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.97504 s, 15.0 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.15468 s, 20.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.605048 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529043 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520131 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325493 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128187 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100387 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128157 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111404 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156545 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146997 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175779 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184707 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163955 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168374 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188429 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212623 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192842 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313661 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700468 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473842 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125178 s, 83.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301762 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550413 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786006 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411979 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544136 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063582 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494125 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994865 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141027 s, 74.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122061 s, 85.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177824 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242157 s, 43.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg630-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg630-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.55186 s, 12.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.08011 s, 20.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (89s) == sanityn test 77o: Changing rank should not panic ====== 13:39:18 (1788543558) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (10s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 13:39:29 (1788543569) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (92s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 13:41:00 (1788543660) mds.MDS.mdt.nrs_policies=tbf oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg630-client: oleg630-server: ssh exited with exit code 36 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg630-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg630-client: oleg630-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (31s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 13:41:31 (1788543691) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273030 kill md_thread xargs PASS 77r (56s) == sanityn test 77s: Check TBF LRU shrinker ============== 13:42:27 (1788543747) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (60s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 13:43:27 (1788543807) oleg630-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg630-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg630-client: oleg630-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (9s) == sanityn test 79: xattr: intent error ================== 13:43:36 (1788543816) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935647 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 13:43:38.000000000 -0400 Modify: 2026-09-04 13:43:38.000000000 -0400 Change: 2026-09-04 13:43:38.000000000 -0400 Birth: 2026-09-04 13:43:37.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (18s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 13:43:54 (1788543834) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 13:43:57 (1788543837) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (2s) == sanityn test 81a: rename and stat under striped directory ========================================================== 13:44:00 (1788543840) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 13:44:02 (1788543842) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 13:44:06 (1788543846) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (2s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 13:44:09 (1788543849) - open/close 707 (time 1788543863.57 total 10.01 last 70.61) - open/close 1307 (time 1788543873.59 total 20.03 last 59.88) - open/close 2051 (time 1788543883.59 total 30.04 last 74.37) total: 2500 open/close in 35.49 seconds: 70.44 ops/second snapshot_time 1788543977.533865405 secs.nsecs start_time 1788543850.220889567 secs.nsecs elapsed_time 127.312975838 secs.nsecs open 2505 samples [usecs] 127 19105 4088564 11712090584 close 2504 samples [usecs] 122 12166 975566 922356980 mknod 2501 samples [usecs] 501 18943 3575105 9439842427 unlink 2500 samples [usecs] 755 103344 5736664 38178160708 mkdir 2 samples [usecs] 885 1693 2578 3649474 rmdir 2 samples [usecs] 11454 18802 30256 484709320 rename 2500 samples [usecs] 458 89687 17594263 264443925919 getattr 3762 samples [usecs] 10 7034 275704 350218160 parallel_rename_file 2500 samples [usecs] 458 89687 17594263 264443925919 crossdir_rename 2500 samples [usecs] 458 89687 17594263 264443925919 readpage 14 samples [usecs] 4251 34744 230863 4935778531 PASS 81d (134s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 13:46:23 (1788543983) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7612 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7612 PASS 82 (6s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 13:46:29 (1788543989) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (3s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 13:46:32 (1788543992) fail_loc=0x8000060b PASS 84 (11s) == sanityn test 85: Lustre API root cache race =========== 13:46:43 (1788544003) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788544004 Finishing test test1 at 1788544009 PASS 85 (10s) == sanityn test 90: open/create and unlink striped directory ========================================================== 13:46:53 (1788544013) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (2s) == sanityn test 91: chmod and unlink striped directory === 13:46:55 (1788544015) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 13:46:58 (1788544018) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (2s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 13:47:01 (1788544021) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5765 0x1685 0x240000400 1 5726 0x165e 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5766 0x1686 0x240000400 1 5727 0x165f 0x280000400 rc1=0 and rc2=0 PASS 93 (15s) == sanityn test 94: signal vs CP callback race =========== 13:47:16 (1788544036) osc.lustre-OST0000-osc-ffff956992743000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9569a0e80000.idle_timeout=0 osc.lustre-OST0001-osc-ffff956992743000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9569a0e80000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.159225 s, 51.4 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.00762 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 289719 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.13479 s, 30.4 kB/s osc.lustre-OST0000-osc-ffff956992743000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9569a0e80000.idle_timeout=debug osc.lustre-OST0001-osc-ffff956992743000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9569a0e80000.idle_timeout=debug PASS 94 (15s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 13:47:31 (1788544051) llite.lustre-ffff956992743000.fast_read=0 llite.lustre-ffff9569a0e80000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0830521 s, 25.3 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff956992743000.fast_read=1 llite.lustre-ffff9569a0e80000.fast_read=1 PASS 95a (20s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 13:47:51 (1788544071) llite.lustre-ffff956992743000.fast_read=0 llite.lustre-ffff9569a0e80000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0057571 s, 2.1 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff956992743000.fast_read=1 llite.lustre-ffff9569a0e80000.fast_read=1 PASS 95b (13s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 13:48:04 (1788544084) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 13:48:07 (1788544087) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00699 s, 586 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (7s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 13:48:15 (1788544095) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.220541 s, 9.5 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (9s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 13:48:23 (1788544103) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.220745 s, 9.5 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (8s) == sanityn test 100e: DoM: read on open and file size ==== 13:48:31 (1788544111) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0161252 s, 744 kB/s PASS 100e (5s) == sanityn test 101a: Discard DoM data on unlink ========= 13:48:36 (1788544116) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00345574 s, 1.2 MB/s PASS 101a (6s) == sanityn test 101b: Discard DoM data on rename ========= 13:48:42 (1788544122) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0221886 s, 185 kB/s PASS 101b (8s) == sanityn test 101c: Discard DoM data on close-unlink === 13:48:50 (1788544130) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0223419 s, 183 kB/s PASS 101c (10s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 13:49:00 (1788544140) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 40 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf40:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (9s) == sanityn test 103: Test size correctness with lockahead ========================================================== 13:49:09 (1788544149) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788544152 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788544156 Finishing test test23 at 1788544156 PASS 103 (13s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 13:49:22 (1788544162) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 13:49:25 (1788544165) fail_loc=0x416 PASS 105 (21s) == sanityn test 106a: Verify the btime via statx() ======= 13:49:46 (1788544186) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 13:49:49 (1788544189) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0143325 s, 71.4 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938183 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 13:49:50.000000000 -0400 Modify: 2026-09-04 13:49:50.000000000 -0400 Change: 2026-09-04 13:49:50.000000000 -0400 Birth: 2026-09-04 13:49:49.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938183 -rw-r--r-- 1 0 0 1788544189 1788544190 1788544190 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938183 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 13:49:50.000000000 -0400 Modify: 2026-09-04 13:49:50.000000000 -0400 Change: 2026-09-04 13:49:50.000000000 -0400 Birth: 2026-09-04 13:49:49.000000000 -0400 1788544190 1024 1 PASS 106b (6s) == sanityn test 106c: Verify statx attributes mask ======= 13:49:55 (1788544195) PASS 106c (6s) == sanityn test 107a: Basic grouplock conflict =========== 13:50:01 (1788544201) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.23114 s, 45.4 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5775 0x168f 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 First grouplock blocks second one PASS 107a (10s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 13:50:12 (1788544212) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.440055 s, 23.8 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5734 0x1666 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7612 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (13s) == sanityn test 108a: lseek: parallel updates ============ 13:50:25 (1788544225) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00672884 s, 1.5 MB/s llite.lustre-ffff956992743000.hybrid_io=0 llite.lustre-ffff9569a0e80000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.73789 s, 1.8 MB/s PASS 108a (12s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 13:50:37 (1788544237) 192.168.206.130@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg630-client.virtnet /mnt/lustre (opts:) 192.168.206.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg630-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg630-client.virtnet: -o user_xattr,flock 192.168.206.130@tcp:/lustre /mnt/lustre2 Starting client: oleg630-client.virtnet: -o user_xattr,flock 192.168.206.130@tcp:/lustre /mnt/lustre 192.168.206.130@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.206.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg630-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg630-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg630-server' oleg630-server: oleg630-server.virtnet: executing load_modules_local oleg630-server: kmemleak disabled oleg630-server: Loading modules from /home/green/git/lustre-release/lustre oleg630-server: detected 4 online CPUs by sysfs oleg630-server: MODOPTS_LIBCFS= oleg630-server: Force libcfs to create 2 CPU partitions oleg630-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg630-client.virtnet: -o user_xattr,flock 192.168.206.130@tcp:/lustre /mnt/lustre2 Starting client: oleg630-client.virtnet: -o user_xattr,flock 192.168.206.130@tcp:/lustre /mnt/lustre 192.168.206.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.206.130@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg630-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg630-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg630-server' oleg630-server: oleg630-server.virtnet: executing load_modules_local oleg630-server: kmemleak disabled oleg630-server: Loading modules from /home/green/git/lustre-release/lustre oleg630-server: detected 4 online CPUs by sysfs oleg630-server: MODOPTS_LIBCFS= oleg630-server: Force libcfs to create 2 CPU partitions oleg630-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg630-client.virtnet: -o user_xattr,flock 192.168.206.130@tcp:/lustre /mnt/lustre Starting client: oleg630-client.virtnet: -o user_xattr,flock 192.168.206.130@tcp:/lustre /mnt/lustre2 PASS 109 (63s) == sanityn test 110: do not grant another lock on resend ========================================================== 13:51:40 (1788544300) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 13:51:42.000000000 -0400 Modify: 2026-09-04 13:51:42.000000000 -0400 Change: 2026-09-04 13:51:42.000000000 -0400 Birth: 2026-09-04 13:51:41.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg630-client.virtnet oleg630-client.virtnet PASS 110 (171s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 13:54:32 (1788544472) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 13:54:36 (1788544476) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 13:54:40 (1788544480) lustre-OST0000-osc-ffff956992a7b800 active. lustre-OST0001-osc-ffff956992a7b800 active. lustre-OST0000-osc-ffff956992a7b800 active. lustre-OST0001-osc-ffff956992a7b800 active. MGC192.168.206.130@tcp active. PASS 113 (8s) == sanityn test 114: implicit default LMV inherit ======== 13:54:48 (1788544488) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (3s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 13:54:51 (1788544491) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 13:54:54 (1788544494) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (5s) == sanityn test 121: trunc append race =================== 13:54:59 (1788544499) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0156704 s, 1.3 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.102256 s, 200 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (11s) == sanityn test 122: directory size is consistent across mounts ========================================================== 13:55:10 (1788544510) PASS 122 (11s) == sanityn test 200: service remains healthy while able to process request ========================================================== 13:55:21 (1788544521) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0974 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1189 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1099 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1217 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1728 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2252 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1935 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0954 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2149 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2124 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1888 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1894 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1602 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2223 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2292 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2656 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3094 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4967 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg630-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff956992a7b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff956992a7b800.ost_server_uuid in FULL state after 0 sec PASS 200 (83s) cleanup: ====================================================== == sanityn test complete, duration 9924 sec ============== 13:56:47 (1788544607) === sanityn: start cleanup 13:56:49 (1788544609) === Stopping clients: oleg630-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg630-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 14:01:20 (1788544880) ===