-----============= acceptance-small: sanityn ============----- Sun Sep 6 22:04:47 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg137-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 22:05:08 (1788746708) === Starting client oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre2 Mount client oleg137-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre2 Started clients oleg137-client.virtnet: 192.168.201.137@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg137-client.virtnet: executing check_config_client /mnt/lustre oleg137-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg137-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8a8148c21000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8a814963b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a8148c21000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a814963b000.idle_timeout=debug disable quota as required oleg137-server: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 22:05:46 (1788746746) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661132 s, 15.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7664] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 22:05:49 (1788746749) PASS 0 (8s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 22:05:56 (1788746756) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (6s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 22:06:02 (1788746762) -rw-r--r-- 1 root root 0 Sep 6 22:06 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (7s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 22:06:10 (1788746770) -rw-r--r-- 1 root root 0 Sep 6 22:06 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 22:06:16 (1788746776) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 22:06:23 (1788746783) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (6s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 22:06:29 (1788746789) total 1 -rwxrwxrwx 1 root root 0 Sep 6 22:06 f2a -rwxrwxrwx 1 root root 0 Sep 6 22:06 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 6 22:06 f2a -rwxrwxrwx 1 root root 0 Sep 6 22:06 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0122585 s, 41.8 kB/s PASS 2e (6s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 22:06:36 (1788746796) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 22:06:39 (1788746799) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.335729 s, 6.2 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (6s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 22:06:45 (1788746805) PASS 3 (6s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 22:06:52 (1788746812) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 22:06:59 (1788746819) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 22:07:06 (1788746826) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (6s) == sanityn test 7: remove of open directory on other node ======================================================================= 22:07:12 (1788746832) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (7s) == sanityn test 8: remove of open special file on other node ==================================================================== 22:07:19 (1788746839) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (8s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 22:07:27 (1788746847) PASS 9a (8s) == sanityn test 9b: append to striped sparse file ======== 22:07:35 (1788746855) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419719 s, 25.0 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0515883 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 22:07:42 (1788746862) 1+0 records in 1+0 records out 1 byte copied, 0.00558644 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0109351 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0148849 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0208607 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0114957 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0189783 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0204995 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0129746 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0175022 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.011457 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0124357 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0115974 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 22:07:50 (1788746870) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0317788 s, 96.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0670805 s, 61.1 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000461747 s, 6.7 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 22:07:57 (1788746877) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7664 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (6s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 22:08:04 (1788746884) warning: '-runtime' deprecated, use '-t runtime' instead - link 394 (time 1788746896.07 total 10.03 last 39.28) - link 784 (time 1788746906.08 total 20.05 last 38.94) - link 1239 (time 1788746916.09 total 30.05 last 45.47) - link 1567 (time 1788746926.11 total 40.07 last 32.75) - link 1889 (time 1788746936.12 total 50.08 last 32.16) - link 2208 (time 1788746946.17 total 60.13 last 31.76) - link 2529 (time 1788746956.18 total 70.14 last 32.05) - link 2924 (time 1788746966.19 total 80.15 last 39.47) - link 3261 (time 1788746976.21 total 90.17 last 33.64) total: 3572 link in 100.03 seconds: 35.71 ops/second using seed 4268991612 running for 100 seconds - stat 6084 (time 1788746898 ; total 11 ; last 11) - stat 10000 (time 1788746903 ; total 16 ; last 5) - stat 17717 (time 1788746914 ; total 27 ; last 11) - stat 20000 (time 1788746916 ; total 29 ; last 2) - stat 25727 (time 1788746927 ; total 40 ; last 11) - stat 30000 (time 1788746934 ; total 47 ; last 7) - stat 33234 (time 1788746945 ; total 58 ; last 11) - stat 36219 (time 1788746956 ; total 69 ; last 11) - stat 40000 (time 1788746966 ; total 79 ; last 10) - stat 42782 (time 1788746977 ; total 90 ; last 11) total: 45652 stats in 100 seconds: 456.519989 stats/second - unlinked 0 (time 1788746937 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3572) error: No such file or directory total: 3571 unlinks in 90 seconds: 39.677776 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21464) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21466) - No such process PASS 12 (147s) == sanityn test 13: test directory page revocation ======= 22:10:31 (1788747031) f13 f13 PASS 13 (7s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 22:10:38 (1788747038) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7664 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 22649 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (6s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 22:10:44 (1788747044) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 23228 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (6s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 22:10:50 (1788747050) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23806 Terminated $DIR1/$tdir/sleep 60 PASS 14b (7s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 22:10:57 (1788747057) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24388 Terminated $DIR1/$tdir/sleep 60 PASS 14c (6s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 22:11:03 (1788747063) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24970 Terminated $DIR1/$tdir/sleep 60 PASS 14d (7s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 22:11:10 (1788747070) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (47s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 22:11:58 (1788747118) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.61336 s, 13.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.445742 s, 18.8 MB/s Chance of close/open is 1 in 50 Seed set to 7122 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x94e945 000002[0] 1788747123.515879 trunc from 00000000 to 0x94e944 (0x94e945 bytes) 000005[1] 1788747123.646811 punch from 0x1e845e to 0x1f68d1 (0xe474 bytes) 000006[0] 1788747123.685476 punch from 0x7ffc23 to 0x804dbd (0x519b bytes) 000007[0] 1788747123.732829 trunc from 0x94e945 to 0x5e585c (0x3690e8 bytes) 000008[0] 1788747123.758668 write 0x998984 thru 0x9a0eb4 (0x8531 bytes) 000010[0] 1788747123.789663 write 0x0aa312 thru 0x0b6000 (0xbcef bytes) 000012[0] 1788747123.804902 trunc from 0x9a0eb5 to 0x4d6ad7 (0x4ca3dd bytes) 000013[1] 1788747123.821949 mapwrite 0x30329f thru 0x311135 (0xde97 bytes) 000015[0] 1788747123.981158 trunc from 0x4d6ad8 to 0x6c74f5 (0x1f0a1e bytes) 000017[1] 1788747124.041412 write 0x9cfefa thru 0x9d1dca (0x1ed1 bytes) 000018[1] 1788747124.070524 read 0x0fea70 thru 0x107ac9 (0x905a bytes) 000019[0] 1788747124.098944 write 0x7ef1b7 thru 0x7f409b (0x4ee5 bytes) 000020[0] 1788747124.111474 trunc from 0x9d1dcb to 0x0f358f (0x8de83b bytes) 000021[1] 1788747124.173041 mapread 0x00fb80 thru 0x01e7f3 (0xec74 bytes) 000022[0] 1788747124.237504 mapread 0x0100f6 thru 0x01fe03 (0xfd0e bytes) 000023[1] 1788747124.286834 trunc from 0x0f3590 to 0x3bf410 (0x2cbe81 bytes) 000024[0] 1788747124.381670 punch from 0x0ae932 to 0x0b529f (0x696e bytes) 000025[1] 1788747124.414152 mapread 0x2c56a2 thru 0x2d1a15 (0xc374 bytes) 000026[1] 1788747124.465161 punch from 0x198684 to 0x1a35da (0xaf57 bytes) 000027[0] 1788747124.495976 trunc from 0x3bf411 to 0x685917 (0x2c6507 bytes) 000028[0] 1788747124.555415 trunc from 0x685918 to 0x6efb99 (0x6a282 bytes) 000029[0] 1788747124.597456 mapwrite 0x695f5a thru 0x6a4d40 (0xede7 bytes) 000030[1] 1788747124.659554 punch from 0x084f0c to 0x08b4f3 (0x65e8 bytes) 000031[0] 1788747124.697753 read 0x03d9b7 thru 0x04d53d (0xfb87 bytes) 000033[0] 1788747124.749609 mapwrite 0x2c9c57 thru 0x2caf37 (0x12e1 bytes) 000034[1] 1788747124.835826 punch from 0x187496 to 0x18ebb1 (0x771c bytes) 000036[0] 1788747124.885545 mapread 0x3751a3 thru 0x37961c (0x447a bytes) 000037[0] 1788747124.925695 punch from 0x527644 to 0x534d5b (0xd718 bytes) 000038[1] 1788747124.953999 write 0x9083b2 thru 0x91122b (0x8e7a bytes) 000039[1] 1788747124.989086 mapwrite 0x28a7e1 thru 0x293d3e (0x955e bytes) 000042[1] 1788747125.134531 mapread 0x84423d thru 0x848686 (0x444a bytes) 000043[0] 1788747125.194918 mapwrite 0x05d9e9 thru 0x065543 (0x7b5b bytes) 000045[0] 1788747125.301140 mapread 0x8b353c thru 0x8ba7b6 (0x727b bytes) 000046[0] 1788747125.339856 mapwrite 0x239dae thru 0x24069d (0x68f0 bytes) 000047[0] 1788747125.424783 read 0x81528c thru 0x821df3 (0xcb68 bytes) 000049[0] 1788747125.507152 punch from 0x609736 to 0x6179f6 (0xe2c1 bytes) 000050[0] 1788747125.516692 mapread 0x400434 thru 0x40bf63 (0xbb30 bytes) 000053[0] 1788747125.586778 mapread 0x8f1067 thru 0x8f31dc (0x2176 bytes) 000054[1] 1788747125.631698 punch from 0x338d9c to 0x348545 (0xf7aa bytes) 000055[1] 1788747125.666803 trunc from 0x91122c to 0x24df6c (0x6c32bf bytes) 000058[0] 1788747125.730725 read 0x1923a2 thru 0x19a2a2 (0x7f01 bytes) 000059[1] 1788747125.807088 trunc from 0x24df6d to 0x4e06fe (0x292792 bytes) 000060[0] 1788747125.841631 write 0x50e8b6 thru 0x5105fb (0x1d46 bytes) 000063[0] 1788747125.946264 trunc from 0x5105fc to 0x49fccb (0x70930 bytes) 000064[0] 1788747125.990723 punch from 0x27bc14 to 0x28a7e5 (0xebd2 bytes) 000066[0] 1788747126.034312 write 0x2cc961 thru 0x2ce8d2 (0x1f72 bytes) 000067[1] 1788747126.093344 read 0x11dba1 thru 0x120916 (0x2d76 bytes) 000070[1] 1788747126.132037 mapwrite 0x1b37bb thru 0x1bfed6 (0xc71c bytes) 000075[1] 1788747126.348995 punch from 0x24b89f to 0x24f4cf (0x3c31 bytes) 000076[1] 1788747126.364682 mapwrite 0x0b9dd2 thru 0x0b9fe4 (0x213 bytes) 000078[1] 1788747126.467798 read 0x3e6098 thru 0x3e6d25 (0xc8e bytes) 000079[1] 1788747126.495517 mapwrite 0x7b1cf2 thru 0x7bf1ed (0xd4fc bytes) 000080[0] 1788747126.677576 mapwrite 0x796d82 thru 0x7a105c (0xa2db bytes) 000082[0] 1788747126.832578 mapread 0x354dc3 thru 0x35df68 (0x91a6 bytes) 000083[1] 1788747126.893101 mapread 0x0a8bbd thru 0x0ae908 (0x5d4c bytes) 000085[0] 1788747126.936558 mapread 0x56a706 thru 0x5768e3 (0xc1de bytes) 000086[1] 1788747126.937409 mapread 0x092c73 thru 0x09c597 (0x9925 bytes) 000088[1] 1788747126.976345 write 0x5258c0 thru 0x531e6b (0xc5ac bytes) 000089[1] 1788747127.035326 punch from 0x78d290 to 0x79a11f (0xce90 bytes) 000091[0] 1788747127.048946 mapwrite 0x0b455a thru 0x0b45b5 (0x5c bytes) 000092[1] 1788747127.152408 read 0x3f65e2 thru 0x403ab8 (0xd4d7 bytes) 000093[0] 1788747127.208126 write 0x6d6ad3 thru 0x6e66cd (0xfbfb bytes) 000095[1] 1788747127.245107 mapwrite 0x12854d thru 0x12f98e (0x7442 bytes) 000097[1] 1788747127.382999 mapread 0x59dea4 thru 0x5abf7f (0xe0dc bytes) 000098[0] 1788747127.444135 read 0x4c903b thru 0x4cf271 (0x6237 bytes) 000103[0] 1788747127.492648 write 0x865287 thru 0x8723b7 (0xd131 bytes) 000104[1] 1788747127.525818 trunc from 0x8723b8 to 0x73d025 (0x135392 bytes) 000105[1] 1788747127.624137 trunc from 0x73d026 to 0x23d03f (0x4fffe6 bytes) 000106[1] 1788747127.681576 read 0x1f95b0 thru 0x1fa59c (0xfed bytes) 000107[0] 1788747127.740383 trunc from 0x23d040 to 0x219307 (0x23d38 bytes) 000109[1] 1788747127.845669 read 0x073bfd thru 0x076eb6 (0x32ba bytes) 000110[0] 1788747127.930854 punch from 0x10a1be to 0x10afd2 (0xe15 bytes) 000111[1] 1788747127.977943 mapread 0x084a05 thru 0x08f5e3 (0xabdf bytes) 000113[0] 1788747128.079438 read 0x03a6f0 thru 0x03e4ab (0x3dbc bytes) 000114[1] 1788747128.153624 punch from 0x19f1b7 to 0x1ac95b (0xd7a5 bytes) 000115[0] 1788747128.207109 punch from 0x20dcbd to 0x21367e (0x59c2 bytes) 000118[1] 1788747128.300557 write 0x424162 thru 0x42ade1 (0x6c80 bytes) 000119[1] 1788747128.365793 trunc from 0x42ade2 to 0x0b4d41 (0x3760a0 bytes) 000120[1] 1788747128.440332 trunc from 0x0b4d42 to 0x933e33 (0x87f0f2 bytes) 000122[0] 1788747128.497932 punch from 0x5f8b10 to 0x60143f (0x8930 bytes) 000123[1] 1788747128.530374 read 0x1ba5ad thru 0x1c922a (0xec7e bytes) 000124[1] 1788747128.558517 mapread 0x420676 thru 0x424689 (0x4014 bytes) 000125[0] 1788747128.628329 punch from 0x5ebd78 to 0x5f752a (0xb7b3 bytes) 000126[0] 1788747128.666244 mapread 0x57a9c3 thru 0x5846f7 (0x9d35 bytes) 000127[0] 1788747128.742265 mapwrite 0x0a44cc thru 0x0ab9ac (0x74e1 bytes) 000128[0] 1788747128.864981 read 0x3b5bb3 thru 0x3bed8d (0x91db bytes) 000129[0] 1788747128.891868 punch from 0x1b65e7 to 0x1b6d71 (0x78b bytes) 000131[1] 1788747128.931302 read 0x6d0cd3 thru 0x6da15f (0x948d bytes) 000133[0] 1788747128.946780 punch from 0x78f3d5 to 0x78fce7 (0x913 bytes) 000135[0] 1788747128.980516 mapread 0x856c6f thru 0x8662db (0xf66d bytes) 000136[0] 1788747129.012967 mapread 0x79b316 thru 0x7a69e0 (0xb6cb bytes) 000138[1] 1788747129.103809 read 0x0d3c27 thru 0x0daa29 (0x6e03 bytes) 000142[0] 1788747129.160316 write 0x300370 thru 0x307f8b (0x7c1c bytes) 000143[0] 1788747129.179896 mapread 0x7273b5 thru 0x729419 (0x2065 bytes) 000144[0] 1788747129.238602 punch from 0x8d64e9 to 0x8de451 (0x7f69 bytes) 000146[0] 1788747129.282509 trunc from 0x933e34 to 0x822df9 (0x11103a bytes) 000150[0] 1788747129.332911 read 0x57f7be thru 0x58c644 (0xce87 bytes) 000151[0] 1788747129.387122 punch from 0x48d008 to 0x48d877 (0x870 bytes) 000153[1] 1788747129.419431 mapread 0x776d07 thru 0x777f02 (0x11fc bytes) 000154[1] 1788747129.466308 punch from 0x38d303 to 0x398011 (0xad0f bytes) 000155[0] 1788747129.528700 mapwrite 0x541aea thru 0x543168 (0x167f bytes) 000157[1] 1788747129.659812 punch from 0x1e8156 to 0x1f2a73 (0xa91e bytes) 000159[1] 1788747129.695639 punch from 0x18a59c to 0x18e612 (0x4077 bytes) 000160[0] 1788747129.716961 read 0x2520e4 thru 0x25f012 (0xcf2f bytes) 000162[0] 1788747129.751564 read 0x7bf19b thru 0x7bfc86 (0xaec bytes) 000163[1] 1788747129.757384 read 0x632460 thru 0x6344b3 (0x2054 bytes) 000166[0] 1788747129.786746 trunc from 0x822dfa to 0x027d44 (0x7fb0b5 bytes) 000168[0] 1788747129.847738 mapread 0x00522d thru 0x007ece (0x2ca2 bytes) 000171[1] 1788747129.957295 write 0x443644 thru 0x447cb5 (0x4672 bytes) 000172[0] 1788747129.996982 trunc from 0x447cb6 to 0x804ea7 (0x3bd1f2 bytes) 000173[1] 1788747130.074973 mapwrite 0x7df869 thru 0x7ef132 (0xf8ca bytes) truncating to largest ever: 0x9fd245 000174[1] 1788747130.247234 trunc from 0x804ea8 to 0x9fd244 (0x1f839d bytes) 000175[1] 1788747130.317840 punch from 0x6f7a52 to 0x6fdcc8 (0x6277 bytes) 000176[0] 1788747130.353738 mapread 0x1f74d7 thru 0x1f7f2f (0xa59 bytes) 000177[0] 1788747130.402267 punch from 0x40c136 to 0x4183b2 (0xc27d bytes) 000179[0] 1788747130.444658 trunc from 0x9fd245 to 0x18c375 (0x870ecf bytes) 000181[0] 1788747130.528258 write 0x59b55a thru 0x5a9782 (0xe229 bytes) 000182[1] 1788747130.562972 read 0x31c23f thru 0x3225a8 (0x636a bytes) 000183[0] 1788747130.634625 mapwrite 0x396a6a thru 0x3a2f5d (0xc4f4 bytes) 000187[1] 1788747130.827318 read 0x18a72d thru 0x196879 (0xc14d bytes) 000189[0] 1788747130.868563 read 0x2202d2 thru 0x222cdb (0x2a0a bytes) 000190[1] 1788747130.893970 trunc from 0x5a9783 to 0x9186ec (0x36ef6a bytes) 000191[0] 1788747130.990706 read 0x1787d7 thru 0x184369 (0xbb93 bytes) 000193[0] 1788747131.044612 trunc from 0x9186ed to 0x4c69f7 (0x451cf5 bytes) 000195[0] 1788747131.090157 punch from 0x0afb13 to 0x0ba879 (0xad67 bytes) 000198[1] 1788747131.152698 mapwrite 0x4aa811 thru 0x4b9a80 (0xf270 bytes) 000199[1] 1788747131.273851 read 0x052320 thru 0x053d21 (0x1a02 bytes) 000201[0] 1788747131.290675 mapwrite 0x6b2497 thru 0x6bc148 (0x9cb2 bytes) 000204[0] 1788747131.393248 mapwrite 0x6e63d2 thru 0x6f1171 (0xada0 bytes) 000207[1] 1788747131.479306 read 0x46458a thru 0x46b999 (0x7410 bytes) 000208[1] 1788747131.523605 punch from 0x6b6b37 to 0x6b6db6 (0x280 bytes) 000209[1] 1788747131.589439 mapwrite 0x4d2c40 thru 0x4df5f9 (0xc9ba bytes) 000210[0] 1788747131.750652 read 0x1c110b thru 0x1c4cdb (0x3bd1 bytes) 000211[0] 1788747131.816745 read 0x380137 thru 0x38d36d (0xd237 bytes) 000213[0] 1788747132.008646 mapread 0x092d50 thru 0x09e84e (0xbaff bytes) 000216[1] 1788747132.099644 punch from 0x1d61f7 to 0x1e07df (0xa5e9 bytes) 000217[1] 1788747132.149380 trunc from 0x6f1172 to 0x5c13c6 (0x12fdab bytes) 000219[0] 1788747132.206504 mapwrite 0x36efe9 thru 0x378edf (0x9ef7 bytes) 000220[1] 1788747132.437027 read 0x0b6d8e thru 0x0c5e7f (0xf0f2 bytes) 000221[0] 1788747132.468144 write 0x53d26f thru 0x549bce (0xc960 bytes) 000222[1] 1788747132.530631 punch from 0x0cb9bb to 0x0cd0b0 (0x16f6 bytes) 000223[1] 1788747132.557805 mapwrite 0x477bc7 thru 0x484431 (0xc86b bytes) 000224[1] 1788747132.669263 trunc from 0x5c13c7 to 0x44b09f (0x176327 bytes) 000225[1] 1788747132.729356 punch from 0x111564 to 0x119c9c (0x8739 bytes) 000226[0] 1788747132.739565 write 0x0627d0 thru 0x06a1b5 (0x79e6 bytes) 000228[1] 1788747132.779002 trunc from 0x44b0a0 to 0x4eec81 (0xa3be2 bytes) 000231[1] 1788747132.842228 punch from 0x2a361b to 0x2ac777 (0x915d bytes) 000232[1] 1788747132.870133 write 0x02df1f thru 0x039581 (0xb663 bytes) 000233[0] 1788747132.882544 write 0x9ee94d thru 0x9f0bdc (0x2290 bytes) 000234[1] 1788747132.929508 trunc from 0x9f0bdd to 0x803f33 (0x1ecca9 bytes) 000235[1] 1788747132.985101 punch from 0x711f95 to 0x71af0e (0x8f7a bytes) 000236[1] 1788747133.012100 write 0x1f39d3 thru 0x20030a (0xc938 bytes) 000237[1] 1788747133.030758 write 0x2322a1 thru 0x23d4fe (0xb25e bytes) 000238[1] 1788747133.041339 write 0x53b0d3 thru 0x543f99 (0x8ec7 bytes) 000239[0] 1788747133.068539 trunc from 0x803f34 to 0x101386 (0x702bad bytes) 000240[1] 1788747133.180043 write 0x47ce93 thru 0x487710 (0xa87e bytes) 000241[1] 1788747133.226474 mapread 0x39cf5e thru 0x39d471 (0x514 bytes) 000242[0] 1788747133.259795 mapread 0x0563d6 thru 0x0619b8 (0xb5e3 bytes) 000243[0] 1788747133.336734 write 0x5d7b0b thru 0x5e4b53 (0xd049 bytes) 000246[0] 1788747133.377754 mapwrite 0x06d18f thru 0x071796 (0x4608 bytes) 000247[0] 1788747133.488331 mapread 0x415446 thru 0x41c9fe (0x75b9 bytes) 000253[0] 1788747133.557918 trunc from 0x5e4b54 to 0x7ac8fa (0x1c7da7 bytes) 000258[1] 1788747133.665281 mapwrite 0x62d3c9 thru 0x62f84d (0x2485 bytes) 000259[0] 1788747133.774784 trunc from 0x7ac8fb to 0x26a8ca (0x542030 bytes) 000260[1] 1788747133.839359 mapread 0x02ac4c thru 0x02ea29 (0x3dde bytes) 000262[1] 1788747133.911705 trunc from 0x26a8cb to 0x9baad4 (0x75020a bytes) 000264[0] 1788747134.036874 trunc from 0x9baad5 to 0x47e53d (0x53c597 bytes) 000266[1] 1788747134.111502 mapread 0x00865b thru 0x01813d (0xfae3 bytes) 000267[0] 1788747134.202024 trunc from 0x47e53e to 0x48c806 (0xe2c9 bytes) 000268[0] 1788747134.292081 write 0x3fc0b6 thru 0x40089d (0x47e8 bytes) 000269[1] 1788747134.344248 mapread 0x43a0a4 thru 0x441ba4 (0x7b01 bytes) 000270[0] 1788747134.413749 read 0x3211ee thru 0x32a226 (0x9039 bytes) 000271[1] 1788747134.457882 punch from 0x0c9445 to 0x0d4d34 (0xb8f0 bytes) 000275[1] 1788747134.547857 punch from 0x11ae6a to 0x126ecb (0xc062 bytes) 000276[0] 1788747134.566531 write 0x765232 thru 0x7661ff (0xfce bytes) 000279[1] 1788747134.623579 mapwrite 0x31b762 thru 0x328ede (0xd77d bytes) 000280[0] 1788747134.747987 write 0x051929 thru 0x058adf (0x71b7 bytes) 000281[1] 1788747134.820516 trunc from 0x766200 to 0x185751 (0x5e0aae bytes) 000283[1] 1788747134.926485 punch from 0x14f6de to 0x151955 (0x2278 bytes) 000284[1] 1788747134.946998 mapwrite 0x97d59c thru 0x980784 (0x31e9 bytes) 000285[1] 1788747135.036917 mapread 0x80f2e4 thru 0x80f93a (0x657 bytes) 000291[1] 1788747135.146281 mapread 0x4d24ac thru 0x4d5219 (0x2d6e bytes) 000293[0] 1788747135.202802 punch from 0x939fcc to 0x93b7ed (0x1822 bytes) 000294[0] 1788747135.218543 punch from 0x758bab to 0x765716 (0xcb6c bytes) 000295[0] 1788747135.249468 write 0x1887ae thru 0x1909fe (0x8251 bytes) 000296[0] 1788747135.284967 mapwrite 0x5626cd thru 0x572388 (0xfcbc bytes) 000297[0] 1788747135.358687 read 0x47b61e thru 0x481ab4 (0x6497 bytes) 000298[0] 1788747135.369895 punch from 0x048cad to 0x057446 (0xe79a bytes) 000304[1] 1788747135.453518 trunc from 0x980785 to 0x21bbec (0x764b98 bytes) 000305[0] 1788747135.588462 mapread 0x141c10 thru 0x14cf1a (0xb30b bytes) 000306[1] 1788747135.665558 trunc from 0x21bbed to 0x92734a (0x70b75e bytes) 000308[0] 1788747135.789793 write 0x4c4ad6 thru 0x4ceef4 (0xa41f bytes) 000312[1] 1788747135.907820 write 0x94501a thru 0x954d1a (0xfd01 bytes) 000313[0] 1788747135.940693 read 0x0f4b8e thru 0x0f7faf (0x3422 bytes) 000315[1] 1788747135.981959 mapwrite 0x29985c thru 0x2a1ab2 (0x8257 bytes) 000316[0] 1788747136.141719 mapwrite 0x8957a6 thru 0x898172 (0x29cd bytes) 000317[1] 1788747136.292087 trunc from 0x954d1b to 0x8bc05d (0x98cbd bytes) 000318[0] 1788747136.386750 write 0x473379 thru 0x4739e2 (0x66a bytes) 000319[0] 1788747136.418813 punch from 0x43929b to 0x4466fb (0xd461 bytes) 000320[1] 1788747136.445061 punch from 0x8461c3 to 0x8540bf (0xdefd bytes) 000321[1] 1788747136.493489 punch from 0x25e7ef to 0x263263 (0x4a75 bytes) 000322[1] 1788747136.522677 punch from 0x58e00b to 0x59a2fd (0xc2f3 bytes) 000323[1] 1788747136.575487 mapwrite 0x4230b8 thru 0x4291e6 (0x612f bytes) 000325[1] 1788747136.685845 read 0x083431 thru 0x08fcbb (0xc88b bytes) 000327[1] 1788747136.700096 punch from 0x533469 to 0x5368a8 (0x3440 bytes) 000329[0] 1788747136.767089 mapread 0x50287c thru 0x506395 (0x3b1a bytes) 000330[1] 1788747136.863139 trunc from 0x8bc05e to 0x5e229e (0x2d9dbf bytes) 000331[0] 1788747136.903921 mapread 0x1c162f thru 0x1c32ac (0x1c7e bytes) 000333[0] 1788747136.978842 read 0x41aa17 thru 0x4224bd (0x7aa7 bytes) 000334[1] 1788747136.998886 mapwrite 0x4bb1aa thru 0x4c7153 (0xbfaa bytes) 000336[1] 1788747137.094155 read 0x5062a1 thru 0x50dbed (0x794d bytes) 000337[1] 1788747137.106299 punch from 0x2fe69c to 0x30dd39 (0xf69e bytes) 000338[0] 1788747137.156099 mapread 0x33d59b thru 0x33fa26 (0x248c bytes) 000339[1] 1788747137.227412 mapwrite 0x075eed thru 0x079726 (0x383a bytes) 000340[0] 1788747137.377389 read 0x097d32 thru 0x09fa4e (0x7d1d bytes) 000341[0] 1788747137.402770 trunc from 0x5e229f to 0x87643d (0x29419f bytes) 000344[0] 1788747137.480947 mapread 0x037e76 thru 0x0440f7 (0xc282 bytes) 000345[0] 1788747137.513149 trunc from 0x87643e to 0x2a69e6 (0x5cfa57 bytes) 000349[1] 1788747137.610687 read 0x076bdf thru 0x07bc83 (0x50a5 bytes) 000350[1] 1788747137.690144 write 0x12f2fa thru 0x138db4 (0x9abb bytes) 000352[1] 1788747137.767727 read 0x1c1ad5 thru 0x1c2658 (0xb84 bytes) 000353[1] 1788747137.864156 punch from 0x0bc753 to 0x0be78c (0x203a bytes) 000356[0] 1788747137.972971 trunc from 0x2a69e7 to 0x530756 (0x289d70 bytes) 000358[1] 1788747138.030090 read 0x37c594 thru 0x38acb1 (0xe71e bytes) 000359[0] 1788747138.114462 punch from 0x217d15 to 0x21ddae (0x609a bytes) 000361[0] 1788747138.146537 mapread 0x3cfab1 thru 0x3db7fe (0xbd4e bytes) 000363[0] 1788747138.179972 mapwrite 0x783fc4 thru 0x793b25 (0xfb62 bytes) 000364[1] 1788747138.267420 mapwrite 0x8a4709 thru 0x8b0502 (0xbdfa bytes) 000366[0] 1788747138.463647 punch from 0x5ffd14 to 0x602762 (0x2a4f bytes) 000367[1] 1788747138.504325 punch from 0x85c8db to 0x85f947 (0x306d bytes) 000371[0] 1788747138.575423 punch from 0x3425ed to 0x351fe9 (0xf9fd bytes) 000374[1] 1788747138.627463 read 0x0ec679 thru 0x0f44e8 (0x7e70 bytes) 000376[0] 1788747138.712367 mapwrite 0x5f05c8 thru 0x5ff250 (0xec89 bytes) 000378[1] 1788747138.867801 trunc from 0x8b0503 to 0x1f7295 (0x6b926d bytes) 000382[0] 1788747139.025638 punch from 0x0a85f3 to 0x0b2e0d (0xa81b bytes) 000383[1] 1788747139.082064 write 0x5fbaba thru 0x600df7 (0x533e bytes) 000386[0] 1788747139.130591 trunc from 0x600df8 to 0x5f8146 (0x8cb1 bytes) 000387[1] 1788747139.174885 trunc from 0x5f8147 to 0x8b57ba (0x2bd674 bytes) 000388[0] 1788747139.217803 trunc from 0x8b57bb to 0x6b99a1 (0x1fbe19 bytes) 000391[1] 1788747139.323657 write 0x9c22d7 thru 0x9d2192 (0xfebc bytes) 000393[1] 1788747139.363742 write 0x4c0dbf thru 0x4c1b04 (0xd46 bytes) 000395[1] 1788747139.423456 mapwrite 0x464c5d thru 0x46bba5 (0x6f49 bytes) 000396[1] 1788747139.493682 punch from 0x56bc9e to 0x56ce6a (0x11cd bytes) 000397[1] 1788747139.499749 mapwrite 0x695b42 thru 0x6a0d32 (0xb1f1 bytes) 000399[0] 1788747139.618002 write 0x73c649 thru 0x745476 (0x8e2e bytes) 000400[0] 1788747139.642542 mapread 0x259e3d thru 0x25b7dc (0x19a0 bytes) 000402[0] 1788747139.694684 read 0x35a3c4 thru 0x3613c7 (0x7004 bytes) 000404[0] 1788747139.718937 trunc from 0x9d2193 to 0x783e52 (0x24e340 bytes) 000407[1] 1788747139.821517 mapwrite 0x835d64 thru 0x839436 (0x36d3 bytes) 000408[1] 1788747140.069201 punch from 0x5bd7e8 to 0x5cb628 (0xde41 bytes) 000409[1] 1788747140.083243 write 0x1f021a thru 0x1f8670 (0x8457 bytes) 000410[1] 1788747140.126959 trunc from 0x839437 to 0x0e0527 (0x758f0f bytes) 000411[0] 1788747140.198076 mapread 0x05c705 thru 0x0659e8 (0x92e4 bytes) 000412[1] 1788747140.283615 mapwrite 0x8bcf64 thru 0x8c23a9 (0x5446 bytes) 000413[1] 1788747140.391693 mapread 0x46369d thru 0x46f372 (0xbcd6 bytes) 000414[1] 1788747140.426472 punch from 0x561d7e to 0x56bdb8 (0xa03b bytes) 000416[0] 1788747140.443482 mapread 0x835923 thru 0x844902 (0xefe0 bytes) 000420[0] 1788747140.536020 punch from 0x1f00ff to 0x1f1a78 (0x197a bytes) 000421[1] 1788747140.607767 write 0x8f2db6 thru 0x8f8989 (0x5bd4 bytes) 000422[1] 1788747140.656488 write 0x190fa4 thru 0x19210a (0x1167 bytes) 000425[0] 1788747140.704966 mapwrite 0x2c8ee6 thru 0x2d7191 (0xe2ac bytes) 000426[0] 1788747140.802240 trunc from 0x8f898a to 0x307c48 (0x5f0d41 bytes) 000428[1] 1788747140.866578 punch from 0x0836fa to 0x087961 (0x4268 bytes) 000429[0] 1788747140.895499 trunc from 0x307c49 to 0x540250 (0x238608 bytes) 000430[1] 1788747140.942877 punch from 0x3836d1 to 0x383a1d (0x34d bytes) 000431[0] 1788747140.983150 mapread 0x06b868 thru 0x070e49 (0x55e2 bytes) 000432[1] 1788747141.045532 punch from 0x08016d to 0x082046 (0x1eda bytes) 000433[1] 1788747141.095780 trunc from 0x540251 to 0x60912c (0xc8edc bytes) 000436[0] 1788747141.179336 mapread 0x436785 thru 0x443b71 (0xd3ed bytes) 000438[1] 1788747141.239050 trunc from 0x60912d to 0x34ca32 (0x2bc6fa bytes) 000439[0] 1788747141.276894 mapwrite 0x2220d6 thru 0x23100c (0xef37 bytes) 000441[0] 1788747141.417393 mapread 0x0fc9bf thru 0x105c68 (0x92aa bytes) 000442[1] 1788747141.465510 read 0x283557 thru 0x28dd8b (0xa835 bytes) 000443[0] 1788747141.610772 mapread 0x1bc64d thru 0x1c3515 (0x6ec9 bytes) 000444[1] 1788747141.635541 mapwrite 0x440e0c thru 0x445455 (0x464a bytes) 000447[0] 1788747141.848517 read 0x2f67bb thru 0x300364 (0x9baa bytes) 000448[1] 1788747141.877703 mapread 0x120c96 thru 0x12c431 (0xb79c bytes) 000451[1] 1788747141.964007 write 0x892684 thru 0x8a1343 (0xecc0 bytes) 000453[0] 1788747142.034456 write 0x045654 thru 0x04f5d0 (0x9f7d bytes) 000456[1] 1788747142.118710 mapwrite 0x729a8e thru 0x73019d (0x6710 bytes) 000457[1] 1788747142.458370 punch from 0x3c47aa to 0x3cc1f1 (0x7a48 bytes) 000458[1] 1788747142.541678 write 0x961545 thru 0x9698bd (0x8379 bytes) 000459[0] 1788747142.556061 write 0x7da233 thru 0x7df102 (0x4ed0 bytes) 000461[0] 1788747142.604314 write 0x422f59 thru 0x430bf5 (0xdc9d bytes) 000462[0] 1788747142.638584 trunc from 0x9698be to 0x66bfbd (0x2fd900 bytes) 000463[0] 1788747142.693542 mapread 0x61f345 thru 0x62ae2f (0xbaeb bytes) 000464[0] 1788747142.733119 read 0x303754 thru 0x3051d3 (0x1a80 bytes) 000466[0] 1788747142.796513 trunc from 0x66bfbe to 0x788a3e (0x11ca81 bytes) 000467[0] 1788747142.837390 trunc from 0x788a3f to 0x9d99f3 (0x250fb5 bytes) 000468[1] 1788747142.876122 mapwrite 0x2534a4 thru 0x25dae7 (0xa644 bytes) 000471[0] 1788747143.070462 mapread 0x8aa9c6 thru 0x8b4ddc (0xa417 bytes) 000475[1] 1788747143.166619 trunc from 0x9d99f4 to 0x072e9d (0x966b56 bytes) 000476[1] 1788747143.244760 read 0x01ab49 thru 0x01fba5 (0x505d bytes) 000477[0] 1788747143.264102 mapread 0x0584ba thru 0x06258a (0xa0d1 bytes) 000478[1] 1788747143.302962 mapread 0x00471a thru 0x0126a7 (0xdf8e bytes) 000480[1] 1788747143.389387 write 0x01c4de thru 0x01e644 (0x2167 bytes) 000483[1] 1788747143.451801 mapwrite 0x078fef thru 0x07f7e9 (0x67fb bytes) 000484[0] 1788747143.547709 punch from 0x059587 to 0x062ed5 (0x994f bytes) 000485[1] 1788747143.585581 mapread 0x058e60 thru 0x0625f9 (0x979a bytes) 000487[1] 1788747143.676804 punch from 0x020ed1 to 0x02133e (0x46e bytes) 000488[1] 1788747143.710553 trunc from 0x07f7ea to 0x46cf8e (0x3ed7a5 bytes) 000489[1] 1788747143.755905 mapwrite 0x2892da thru 0x291d13 (0x8a3a bytes) 000490[1] 1788747143.849536 punch from 0x40c284 to 0x41af42 (0xecbf bytes) 000491[0] 1788747143.872592 write 0x2ae615 thru 0x2b91db (0xabc7 bytes) 000492[0] 1788747143.921606 punch from 0x425e65 to 0x42e1eb (0x8387 bytes) 000494[1] 1788747144.017356 punch from 0x3b7828 to 0x3c3238 (0xba11 bytes) 000495[1] 1788747144.063055 punch from 0x17bbd8 to 0x187291 (0xb6ba bytes) 000497[0] 1788747144.096131 trunc from 0x46cf8f to 0x67f9b8 (0x212a2a bytes) 000498[1] 1788747144.126828 write 0x079426 thru 0x084caa (0xb885 bytes) 000499[0] 1788747144.168839 mapread 0x5345e9 thru 0x53962f (0x5047 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 7144 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1ef5f4 000001[0] 1788747145.097395 trunc from 00000000 to 0x1ef5f3 (0x1ef5f4 bytes) 000002[1] 1788747145.125304 mapwrite 0x6c7000 thru 0x6c72ba (0x2bb bytes) 000006[1] 1788747145.330021 trunc from 0x6c72bb to 0x00d7e3 (0x6b9ad7 bytes) 000008[0] 1788747145.383412 mapwrite 0x18f000 thru 0x19d22c (0xe22d bytes) 000009[0] 1788747145.659615 write_OD 0x81d000 thru 0x82afff (0xe000 bytes) 000011[0] 1788747145.776499 read_OD 0x796000 thru 0x79cfff (0x7000 bytes) truncating to largest ever: 0x939e67 000013[0] 1788747145.793540 trunc from 0x82b000 to 0x939e66 (0x10ee67 bytes) 000014[1] 1788747145.854398 mapwrite 0x07e000 thru 0x088f83 (0xaf84 bytes) 000015[1] 1788747145.946374 write 0x92b000 thru 0x93785c (0xc85d bytes) truncating to largest ever: 0x95d3ed 000016[0] 1788747145.958270 trunc from 0x939e67 to 0x95d3ec (0x23586 bytes) 000017[0] 1788747146.047376 mapwrite 0x7f7000 thru 0x8051f2 (0xe1f3 bytes) truncating to largest ever: 0x9ee38a 000018[0] 1788747146.189369 trunc from 0x95d3ed to 0x9ee389 (0x90f9d bytes) 000019[1] 1788747146.216384 mapread 0x2e8000 thru 0x2f7561 (0xf562 bytes) 000023[0] 1788747146.287485 mapwrite 0x565000 thru 0x56553c (0x53d bytes) 000025[1] 1788747146.449806 write 0x8bb000 thru 0x8c6db2 (0xbdb3 bytes) 000026[0] 1788747146.486681 punch from 0x7980e3 to 0x7a08cc (0x87ea bytes) 000027[0] 1788747146.513576 write_OD 0x2dc000 thru 0x2defff (0x3000 bytes) 000031[1] 1788747146.564421 mapwrite 0x653000 thru 0x6596b7 (0x66b8 bytes) 000032[1] 1788747146.682507 mapwrite 0x1e4000 thru 0x1e60b4 (0x20b5 bytes) 000033[0] 1788747146.752512 write_OD 0x8f7000 thru 0x905fff (0xf000 bytes) 000037[1] 1788747146.829940 trunc from 0x9ee38a to 0x22aca9 (0x7c36e0 bytes) 000039[0] 1788747146.894901 mapread 0x12e000 thru 0x136950 (0x8951 bytes) 000040[0] 1788747146.939899 write_OD 0x906000 thru 0x908fff (0x3000 bytes) 000043[0] 1788747147.001647 punch from 0x19275d to 0x1a0507 (0xddab bytes) 000044[1] 1788747147.063723 punch from 0x054d13 to 0x0636da (0xe9c8 bytes) 000045[0] 1788747147.126798 write_OD 0x4af000 thru 0x4b4fff (0x6000 bytes) 000047[1] 1788747147.203589 mapwrite 0x861000 thru 0x86f0ac (0xe0ad bytes) 000048[1] 1788747147.306137 mapwrite 0x95c000 thru 0x96571a (0x971b bytes) 000049[1] 1788747147.459968 read 0x00c000 thru 0x01b741 (0xf742 bytes) 000050[0] 1788747147.516650 mapwrite 0x526000 thru 0x532b2d (0xcb2e bytes) 000051[1] 1788747147.620991 mapwrite 0x05b000 thru 0x05ee40 (0x3e41 bytes) 000052[0] 1788747147.686899 punch from 0x21ba90 to 0x22222a (0x679b bytes) 000053[0] 1788747147.702270 trunc from 0x96571b to 0x2e4289 (0x681491 bytes) 000056[1] 1788747147.734535 read 0x067000 thru 0x074442 (0xd443 bytes) 000058[0] 1788747147.790692 read_OD 0x10a000 thru 0x117fff (0xe000 bytes) 000061[1] 1788747147.796818 punch from 0x1b8853 to 0x1bbea0 (0x364e bytes) 000062[1] 1788747147.834700 mapread 0x2b2000 thru 0x2bc4e7 (0xa4e8 bytes) 000063[1] 1788747147.932646 write 0x7c8000 thru 0x7ceef7 (0x6ef8 bytes) 000064[1] 1788747147.951353 punch from 0x3ded14 to 0x3ec886 (0xdb73 bytes) 000068[1] 1788747148.015570 trunc from 0x7ceef8 to 0x339876 (0x495681 bytes) 000070[0] 1788747148.067040 mapread 0x307000 thru 0x309b4c (0x2b4d bytes) 000071[0] 1788747148.205381 mapread 0x2a3000 thru 0x2a828e (0x528f bytes) 000074[0] 1788747148.212598 mapwrite 0x439000 thru 0x4442c5 (0xb2c6 bytes) 000076[0] 1788747148.348755 punch from 0x0fbef9 to 0x107720 (0xb828 bytes) 000077[0] 1788747148.366942 mapread 0x43c000 thru 0x43e043 (0x2044 bytes) 000078[0] 1788747148.367780 mapread 0x277000 thru 0x27f0ff (0x8100 bytes) 000079[1] 1788747148.421541 read 0x373000 thru 0x3781a1 (0x51a2 bytes) 000080[0] 1788747148.447456 punch from 0x1799f9 to 0x181e94 (0x849c bytes) 000081[1] 1788747148.461260 mapread 0x092000 thru 0x099548 (0x7549 bytes) 000082[0] 1788747148.496544 punch from 0x039443 to 0x047a56 (0xe614 bytes) 000084[0] 1788747148.513554 punch from 0x1bb00d to 0x1c70d0 (0xc0c4 bytes) 000085[0] 1788747148.517003 write_OD 0x64d000 thru 0x659fff (0xd000 bytes) 000087[0] 1788747148.571683 write_OD 0x13d000 thru 0x13efff (0x2000 bytes) 000088[1] 1788747148.616659 write 0x10a000 thru 0x1158e8 (0xb8e9 bytes) 000091[0] 1788747148.653748 mapwrite 0x1e3000 thru 0x1eb157 (0x8158 bytes) 000093[1] 1788747148.784957 write 0x213000 thru 0x216b9b (0x3b9c bytes) 000094[1] 1788747148.816083 write 0x238000 thru 0x23dc6b (0x5c6c bytes) 000095[1] 1788747148.832601 mapread 0x56e000 thru 0x5708b1 (0x28b2 bytes) 000097[0] 1788747148.863670 mapread 0x26b000 thru 0x26daac (0x2aad bytes) 000098[0] 1788747148.911944 mapread 0x069000 thru 0x074226 (0xb227 bytes) 000100[1] 1788747148.981378 trunc from 0x65a000 to 0x6c7f22 (0x6df23 bytes) 000101[0] 1788747149.025936 punch from 0x63b856 to 0x63ff60 (0x470b bytes) 000102[0] 1788747149.038697 mapwrite 0x220000 thru 0x2264da (0x64db bytes) 000103[0] 1788747149.118353 trunc from 0x6c7f23 to 0x456d08 (0x27121a bytes) 000105[0] 1788747149.147666 mapwrite 0x711000 thru 0x714399 (0x339a bytes) 000107[0] 1788747149.275625 mapwrite 0x1d6000 thru 0x1dc64b (0x664c bytes) 000108[0] 1788747149.350856 read_OD 0x365000 thru 0x36afff (0x6000 bytes) 000110[1] 1788747149.368101 punch from 0x030f5b to 0x03b097 (0xa13d bytes) 000115[1] 1788747149.408466 read 0x39d000 thru 0x3a5253 (0x8254 bytes) 000116[1] 1788747149.415582 mapwrite 0x6e4000 thru 0x6eaa7b (0x6a7c bytes) 000118[1] 1788747149.490930 mapwrite 0x370000 thru 0x377885 (0x7886 bytes) 000123[0] 1788747149.588309 punch from 0x0363c8 to 0x0422b5 (0xbeee bytes) 000124[0] 1788747149.611986 mapread 0x211000 thru 0x22001e (0xf01f bytes) 000125[1] 1788747149.664438 read 0x66c000 thru 0x66d4eb (0x14ec bytes) 000126[0] 1788747149.698707 read_OD 0x58e000 thru 0x59bfff (0xe000 bytes) 000127[0] 1788747149.709267 trunc from 0x71439a to 0x70c278 (0x8121 bytes) 000129[1] 1788747149.787601 punch from 0x431467 to 0x4315a4 (0x13e bytes) 000130[0] 1788747149.822667 punch from 0x400c9f to 0x40727f (0x65e1 bytes) 000132[0] 1788747149.883041 mapread 0x51e000 thru 0x52431b (0x631c bytes) 000134[0] 1788747149.953382 mapwrite 0x331000 thru 0x33620d (0x520e bytes) 000135[1] 1788747150.094367 punch from 0x32eff9 to 0x32f5d6 (0x5de bytes) 000136[0] 1788747150.160542 punch from 0x5c151d to 0x5c22ce (0xdb2 bytes) 000137[0] 1788747150.204044 read_OD 0x3fd000 thru 0x400fff (0x4000 bytes) 000138[0] 1788747150.222497 mapread 0x412000 thru 0x417312 (0x5313 bytes) 000139[0] 1788747150.237941 read_OD 0x349000 thru 0x355fff (0xd000 bytes) 000142[0] 1788747150.246139 punch from 0x4a80aa to 0x4b461e (0xc575 bytes) 000143[0] 1788747150.276765 mapread 0x486000 thru 0x493d05 (0xdd06 bytes) 000148[0] 1788747150.331373 read_OD 0x6e5000 thru 0x6ebfff (0x7000 bytes) 000151[0] 1788747150.356699 read_OD 0x48b000 thru 0x48dfff (0x3000 bytes) 000152[1] 1788747150.361255 read 0x165000 thru 0x169a73 (0x4a74 bytes) 000153[1] 1788747150.371943 punch from 0x0f8727 to 0x107716 (0xeff0 bytes) 000154[1] 1788747150.394761 mapread 0x641000 thru 0x6457c0 (0x47c1 bytes) 000156[0] 1788747150.424575 read_OD 0x29a000 thru 0x29ffff (0x6000 bytes) 000157[1] 1788747150.452427 trunc from 0x70c279 to 0x9b29f7 (0x2a677f bytes) 000158[0] 1788747150.504155 mapwrite 0x1e6000 thru 0x1e6936 (0x937 bytes) 000160[1] 1788747150.565632 read 0x586000 thru 0x587c57 (0x1c58 bytes) 000161[1] 1788747150.593562 read 0x3a9000 thru 0x3ac63c (0x363d bytes) 000163[1] 1788747150.604873 write 0x990000 thru 0x99fe77 (0xfe78 bytes) 000165[0] 1788747150.647491 mapread 0x63c000 thru 0x648c2f (0xcc30 bytes) 000166[0] 1788747150.715929 mapwrite 0x231000 thru 0x240aad (0xfaae bytes) 000167[0] 1788747150.842832 mapread 0x5e6000 thru 0x5f40de (0xe0df bytes) 000168[1] 1788747150.885459 mapread 0x604000 thru 0x613dcb (0xfdcc bytes) 000169[0] 1788747150.929670 trunc from 0x9b29f8 to 0x9b2972 (0x85 bytes) 000171[1] 1788747150.962923 mapwrite 0x444000 thru 0x447662 (0x3663 bytes) 000177[0] 1788747151.077338 write_OD 0x028000 thru 0x030fff (0x9000 bytes) 000179[0] 1788747151.143873 mapread 0x00c000 thru 0x01080c (0x480d bytes) 000180[1] 1788747151.165380 mapread 0x23e000 thru 0x24ad8f (0xcd90 bytes) 000181[0] 1788747151.194969 read_OD 0x947000 thru 0x949fff (0x3000 bytes) 000185[0] 1788747151.201684 trunc from 0x9b2973 to 0x0555b4 (0x95d3be bytes) 000187[1] 1788747151.241590 mapread 0x02b000 thru 0x035305 (0xa306 bytes) 000190[0] 1788747151.282924 mapread 0x012000 thru 0x0138c5 (0x18c6 bytes) 000191[1] 1788747151.298650 mapwrite 0x20b000 thru 0x2130f9 (0x80fa bytes) 000196[1] 1788747151.603469 write 0x2d1000 thru 0x2d6c54 (0x5c55 bytes) 000197[0] 1788747151.629595 trunc from 0x2d6c55 to 0x8ce477 (0x5f7823 bytes) 000198[0] 1788747151.726875 read_OD 0x6de000 thru 0x6dffff (0x2000 bytes) 000199[0] 1788747151.761885 read_OD 0x209000 thru 0x20efff (0x6000 bytes) 000200[0] 1788747151.782050 punch from 0x80e079 to 0x815122 (0x70aa bytes) 000202[1] 1788747151.875881 write 0x381000 thru 0x387841 (0x6842 bytes) 000204[0] 1788747151.958648 trunc from 0x8ce478 to 0x1b5cc7 (0x7187b0 bytes) 000208[1] 1788747152.030959 read 0x0db000 thru 0x0e9fc2 (0xefc3 bytes) 000209[1] 1788747152.112927 mapwrite 0x3a1000 thru 0x3a191c (0x91d bytes) 000210[0] 1788747152.290060 mapread 0x2a6000 thru 0x2a72d9 (0x12da bytes) 000211[0] 1788747152.312043 mapwrite 0x8d7000 thru 0x8db450 (0x4451 bytes) 000213[1] 1788747152.460498 write 0x95f000 thru 0x96799d (0x899e bytes) 000215[1] 1788747152.501442 mapwrite 0x1b3000 thru 0x1bbfe6 (0x8fe7 bytes) 000216[1] 1788747152.605577 mapread 0x04f000 thru 0x057dd1 (0x8dd2 bytes) 000217[1] 1788747152.644321 mapwrite 0x927000 thru 0x934eb9 (0xdeba bytes) 000219[0] 1788747152.829505 write_OD 0x60b000 thru 0x619fff (0xf000 bytes) 000220[1] 1788747152.963363 read 0x188000 thru 0x18e500 (0x6501 bytes) 000226[0] 1788747152.982534 punch from 0x51b488 to 0x523eb9 (0x8a32 bytes) 000227[0] 1788747153.000998 write_OD 0x2aa000 thru 0x2b0fff (0x7000 bytes) 000228[1] 1788747153.079511 write 0x9a5000 thru 0x9a55e4 (0x5e5 bytes) 000231[1] 1788747153.162434 write 0x758000 thru 0x767dcb (0xfdcc bytes) 000234[1] 1788747153.224235 mapwrite 0x7cb000 thru 0x7cd3fc (0x23fd bytes) 000235[0] 1788747153.394100 trunc from 0x9a55e5 to 0x1acb81 (0x7f8a63 bytes) 000238[0] 1788747153.482773 mapwrite 0x4f7000 thru 0x4f9c51 (0x2c52 bytes) 000239[1] 1788747153.591976 punch from 0x0f0fdb to 0x0f8ca1 (0x7cc7 bytes) 000240[1] 1788747153.607556 trunc from 0x4f9c52 to 0x4fbd6e (0x211d bytes) 000242[1] 1788747153.641546 punch from 0x41b594 to 0x4271b9 (0xbc26 bytes) 000244[0] 1788747153.658409 trunc from 0x4fbd6f to 0x1563b4 (0x3a59ba bytes) 000245[1] 1788747153.712326 punch from 0x08d683 to 0x090717 (0x3095 bytes) 000246[1] 1788747153.755489 read 0x121000 thru 0x125323 (0x4324 bytes) 000248[1] 1788747153.834681 read 0x08f000 thru 0x098d4b (0x9d4c bytes) 000250[0] 1788747153.835954 punch from 0x09520b to 0x09741b (0x2211 bytes) 000253[0] 1788747153.860117 mapwrite 0x5b6000 thru 0x5c061c (0xa61d bytes) 000254[1] 1788747153.958085 mapwrite 0x499000 thru 0x49b371 (0x2372 bytes) 000255[0] 1788747154.187683 trunc from 0x5c061d to 0x4a6420 (0x11a1fc bytes) 000256[0] 1788747154.254847 trunc from 0x4a6421 to 0x7eab1a (0x3446fa bytes) 000257[1] 1788747154.275842 punch from 0x6ab8ab to 0x6ba385 (0xeadb bytes) 000258[1] 1788747154.303538 mapread 0x551000 thru 0x55f8cf (0xe8d0 bytes) 000259[0] 1788747154.388042 trunc from 0x7eab1b to 0x960d2b (0x176211 bytes) 000262[0] 1788747154.452339 mapread 0x14b000 thru 0x14fa21 (0x4a22 bytes) 000263[1] 1788747154.491811 trunc from 0x960d2c to 0x160bdb (0x800150 bytes) 000264[1] 1788747154.525820 write 0x275000 thru 0x279cc1 (0x4cc2 bytes) 000265[0] 1788747154.527834 read_OD 0x260000 thru 0x266fff (0x7000 bytes) 000266[0] 1788747154.562273 write_OD 0x06a000 thru 0x06cfff (0x3000 bytes) 000268[0] 1788747154.621595 punch from 0x1f75c7 to 0x204efa (0xd934 bytes) 000270[1] 1788747154.667341 punch from 0x069e8d to 0x073c82 (0x9df6 bytes) 000271[0] 1788747154.684679 trunc from 0x279cc2 to 0x85ab7f (0x5e0ebe bytes) 000273[1] 1788747154.720665 mapwrite 0x211000 thru 0x215366 (0x4367 bytes) 000274[1] 1788747154.803603 mapwrite 0x523000 thru 0x529e02 (0x6e03 bytes) 000277[0] 1788747154.896145 write_OD 0x5e9000 thru 0x5ecfff (0x4000 bytes) 000278[1] 1788747154.944235 mapread 0x39d000 thru 0x3a9d57 (0xcd58 bytes) 000279[1] 1788747155.010304 mapwrite 0x9bd000 thru 0x9c06d0 (0x36d1 bytes) 000282[1] 1788747155.185752 read 0x0ae000 thru 0x0afdd5 (0x1dd6 bytes) 000284[0] 1788747155.222780 read_OD 0x229000 thru 0x234fff (0xc000 bytes) 000288[1] 1788747155.272611 write 0x32b000 thru 0x336328 (0xb329 bytes) 000292[0] 1788747155.336668 read_OD 0x1db000 thru 0x1e8fff (0xe000 bytes) 000294[0] 1788747155.373871 write_OD 0x6bf000 thru 0x6c0fff (0x2000 bytes) 000298[0] 1788747155.423956 punch from 0x6f7c87 to 0x7000c1 (0x843b bytes) 000299[1] 1788747155.454771 trunc from 0x9c06d1 to 0x091d39 (0x92e997 bytes) 000304[1] 1788747155.530506 punch from 0x049a4f to 0x04f8cf (0x5e81 bytes) 000308[0] 1788747155.577187 write_OD 0x4dc000 thru 0x4eafff (0xf000 bytes) 000311[0] 1788747155.636753 read_OD 0x38a000 thru 0x38bfff (0x2000 bytes) 000313[0] 1788747155.641214 read_OD 0x264000 thru 0x267fff (0x4000 bytes) 000314[0] 1788747155.647794 mapwrite 0x573000 thru 0x57e091 (0xb092 bytes) 000317[0] 1788747155.773153 read_OD 0x0fb000 thru 0x104fff (0xa000 bytes) 000318[0] 1788747155.798341 punch from 0x29d9bd to 0x2abc80 (0xe2c4 bytes) 000319[1] 1788747155.815517 write 0x267000 thru 0x276528 (0xf529 bytes) 000320[1] 1788747155.830734 read 0x1e1000 thru 0x1e61a6 (0x51a7 bytes) 000322[1] 1788747155.839449 trunc from 0x57e092 to 0x7e7619 (0x269588 bytes) 000324[0] 1788747155.884983 trunc from 0x7e761a to 0x6323e6 (0x1b5233 bytes) 000325[0] 1788747155.926445 mapwrite 0x48c000 thru 0x491a7f (0x5a80 bytes) 000326[0] 1788747156.000020 trunc from 0x6323e7 to 0x4b6899 (0x17bb4d bytes) 000327[1] 1788747156.027781 write 0x7d3000 thru 0x7d3a0c (0xa0d bytes) 000328[1] 1788747156.068710 punch from 0x47bcf7 to 0x488c08 (0xcf12 bytes) 000329[0] 1788747156.074388 punch from 0x40645c to 0x40b79e (0x5343 bytes) 000330[1] 1788747156.110965 write 0x28b000 thru 0x28da29 (0x2a2a bytes) 000331[0] 1788747156.130786 mapwrite 0x57b000 thru 0x589bd5 (0xebd6 bytes) 000333[1] 1788747156.253731 mapwrite 0x67c000 thru 0x685f43 (0x9f44 bytes) 000335[1] 1788747156.349870 write 0x884000 thru 0x88b90a (0x790b bytes) 000336[1] 1788747156.366278 read 0x1a7000 thru 0x1b03b6 (0x93b7 bytes) 000337[0] 1788747156.376557 punch from 0x305b33 to 0x30bc60 (0x612e bytes) 000338[0] 1788747156.418115 read_OD 0x37a000 thru 0x37afff (0x1000 bytes) 000340[1] 1788747156.437786 read 0x0a9000 thru 0x0ae4e4 (0x54e5 bytes) 000341[1] 1788747156.445947 punch from 0x80a0fa to 0x81697a (0xc881 bytes) 000343[0] 1788747156.496074 punch from 0x7a4618 to 0x7b04e5 (0xbece bytes) 000344[1] 1788747156.506682 trunc from 0x88b90b to 0x33f965 (0x54bfa5 bytes) 000345[0] 1788747156.526127 mapread 0x2a3000 thru 0x2a3735 (0x736 bytes) 000346[0] 1788747156.560784 mapwrite 0x8fb000 thru 0x8fcaf8 (0x1af9 bytes) 000348[0] 1788747156.705548 read_OD 0x73a000 thru 0x73efff (0x5000 bytes) 000350[1] 1788747156.739256 mapread 0x2be000 thru 0x2c0257 (0x2258 bytes) 000351[0] 1788747156.757529 mapread 0x5c8000 thru 0x5d3d20 (0xbd21 bytes) 000352[0] 1788747156.785428 trunc from 0x8fcaf9 to 0x807efa (0xf4bfe bytes) 000353[1] 1788747156.822794 write 0x8d0000 thru 0x8de764 (0xe765 bytes) 000354[1] 1788747156.852300 write 0x25e000 thru 0x26d4e2 (0xf4e3 bytes) 000355[0] 1788747156.866256 mapwrite 0x836000 thru 0x843990 (0xd991 bytes) 000356[0] 1788747157.002903 trunc from 0x8de765 to 0x63076a (0x2adffa bytes) 000359[1] 1788747157.082821 read 0x35f000 thru 0x36041d (0x141e bytes) 000360[0] 1788747157.097670 punch from 0x263ab3 to 0x26c82d (0x8d7b bytes) 000361[0] 1788747157.113489 mapwrite 0x18b000 thru 0x18c68a (0x168b bytes) 000362[0] 1788747157.187807 mapread 0x5df000 thru 0x5e0ccf (0x1cd0 bytes) 000364[0] 1788747157.201641 mapwrite 0x01e000 thru 0x01f4a1 (0x14a2 bytes) 000366[1] 1788747157.320312 write 0x8d1000 thru 0x8d1528 (0x529 bytes) 000367[1] 1788747157.351895 mapread 0x875000 thru 0x878638 (0x3639 bytes) 000368[1] 1788747157.384559 mapwrite 0x35c000 thru 0x35f4e7 (0x34e8 bytes) 000369[1] 1788747157.452110 trunc from 0x8d1529 to 0x04d4ce (0x88405a bytes) 000370[1] 1788747157.474236 mapread 0x002000 thru 0x00bfa1 (0x9fa2 bytes) 000371[0] 1788747157.509927 trunc from 0x04d4cf to 0x690861 (0x643393 bytes) 000373[0] 1788747157.588092 mapwrite 0x91c000 thru 0x9281b5 (0xc1b6 bytes) 000375[1] 1788747157.693948 punch from 0x61339b to 0x622263 (0xeec9 bytes) 000377[0] 1788747157.708938 mapread 0x591000 thru 0x5a0c38 (0xfc39 bytes) 000378[0] 1788747157.753636 write_OD 0x869000 thru 0x872fff (0xa000 bytes) 000379[1] 1788747157.806526 read 0x54d000 thru 0x55ac10 (0xdc11 bytes) 000380[1] 1788747157.828353 mapwrite 0x5c7000 thru 0x5cbcd3 (0x4cd4 bytes) 000381[0] 1788747157.916431 mapread 0x6d3000 thru 0x6d81ad (0x51ae bytes) 000384[1] 1788747157.962297 trunc from 0x9281b6 to 0x3c8a49 (0x55f76c bytes) 000386[0] 1788747158.000920 punch from 0x0fa7bd to 0x106700 (0xbf44 bytes) 000387[0] 1788747158.020502 trunc from 0x3c8a4a to 0x39b1f6 (0x2d853 bytes) 000390[1] 1788747158.074548 read 0x298000 thru 0x29cdfd (0x4dfe bytes) 000391[1] 1788747158.161917 trunc from 0x39b1f7 to 0x2d7ab6 (0xc3740 bytes) 000394[1] 1788747158.233027 mapwrite 0x4f4000 thru 0x4f7271 (0x3272 bytes) 000395[1] 1788747158.357533 trunc from 0x4f7272 to 0x33fc61 (0x1b7610 bytes) 000396[0] 1788747158.373784 trunc from 0x33fc62 to 0x52c272 (0x1ec611 bytes) 000399[0] 1788747158.427795 mapwrite 0x967000 thru 0x96d349 (0x634a bytes) 000400[1] 1788747158.528081 mapwrite 0x2d2000 thru 0x2d83d4 (0x63d5 bytes) 000401[1] 1788747158.630604 punch from 0x266714 to 0x26eb85 (0x8472 bytes) 000403[1] 1788747158.646109 read 0x4d6000 thru 0x4df109 (0x910a bytes) 000404[0] 1788747158.673293 mapread 0x6f6000 thru 0x6f944d (0x344e bytes) 000405[0] 1788747158.713635 mapread 0x440000 thru 0x446d0b (0x6d0c bytes) 000406[1] 1788747158.733089 mapwrite 0x338000 thru 0x33d961 (0x5962 bytes) 000410[0] 1788747158.841923 trunc from 0x96d34a to 0x50d645 (0x45fd04 bytes) 000411[1] 1788747158.885458 mapwrite 0x57a000 thru 0x58156b (0x756c bytes) 000412[0] 1788747158.970430 punch from 0x187e4b to 0x19485d (0xca13 bytes) 000413[1] 1788747159.004267 read 0x179000 thru 0x18131a (0x831b bytes) 000416[1] 1788747159.030537 mapread 0x254000 thru 0x261e7b (0xde7c bytes) 000417[0] 1788747159.066694 read_OD 0x3d5000 thru 0x3e2fff (0xe000 bytes) 000418[0] 1788747159.083525 read_OD 0x2fd000 thru 0x2fdfff (0x1000 bytes) 000420[0] 1788747159.088749 punch from 0x2b9591 to 0x2c5284 (0xbcf4 bytes) 000421[0] 1788747159.110112 trunc from 0x58156c to 0x58577d (0x4212 bytes) 000422[1] 1788747159.147868 mapread 0x128000 thru 0x132e92 (0xae93 bytes) 000425[1] 1788747159.203812 write 0x489000 thru 0x48fd1a (0x6d1b bytes) 000426[0] 1788747159.252541 trunc from 0x58577e to 0x19e3e7 (0x3e7396 bytes) 000427[1] 1788747159.322827 mapread 0x035000 thru 0x0376e8 (0x26e9 bytes) 000428[0] 1788747159.359078 mapwrite 0x3c0000 thru 0x3c9e8f (0x9e90 bytes) 000430[0] 1788747159.520424 mapwrite 0x094000 thru 0x099238 (0x5239 bytes) 000431[1] 1788747159.594866 punch from 0x1cfdd5 to 0x1d77c8 (0x79f4 bytes) 000432[0] 1788747159.627912 punch from 0x1fce84 to 0x20a84c (0xd9c9 bytes) 000435[1] 1788747159.676159 mapwrite 0x9af000 thru 0x9b062c (0x162d bytes) 000436[0] 1788747159.798868 mapwrite 0x908000 thru 0x90dcbd (0x5cbe bytes) 000439[0] 1788747159.921988 punch from 0x7b8935 to 0x7bcf91 (0x465d bytes) 000441[1] 1788747159.941520 mapread 0x71b000 thru 0x71b161 (0x162 bytes) 000446[1] 1788747159.967315 read 0x426000 thru 0x433353 (0xd354 bytes) 000448[1] 1788747159.975450 mapread 0x7bf000 thru 0x7bf967 (0x968 bytes) 000449[0] 1788747159.982815 read_OD 0x4d1000 thru 0x4d3fff (0x3000 bytes) 000451[0] 1788747159.987989 read_OD 0x846000 thru 0x847fff (0x2000 bytes) 000452[1] 1788747159.992668 write 0x275000 thru 0x28470e (0xf70f bytes) 000454[0] 1788747160.035542 mapwrite 0x2da000 thru 0x2dcc18 (0x2c19 bytes) 000457[0] 1788747160.129678 trunc from 0x9b062d to 0x605f55 (0x3aa6d7 bytes) 000458[0] 1788747160.142533 mapwrite 0x90b000 thru 0x90b897 (0x898 bytes) 000459[1] 1788747160.218753 mapwrite 0x023000 thru 0x02c693 (0x9694 bytes) 000462[0] 1788747160.350721 read_OD 0x284000 thru 0x292fff (0xf000 bytes) 000464[0] 1788747160.373326 read_OD 0x62c000 thru 0x639fff (0xe000 bytes) 000467[1] 1788747160.385261 mapwrite 0x563000 thru 0x5665c7 (0x35c8 bytes) 000471[1] 1788747160.480390 write 0x2db000 thru 0x2eadd1 (0xfdd2 bytes) 000472[1] 1788747160.487429 mapwrite 0x84d000 thru 0x84f08a (0x208b bytes) 000475[1] 1788747160.600861 read 0x2dc000 thru 0x2ebbfc (0xfbfd bytes) 000478[1] 1788747160.637637 mapread 0x7fd000 thru 0x80704c (0xa04d bytes) 000479[0] 1788747160.655761 read_OD 0x6bf000 thru 0x6c8fff (0xa000 bytes) 000481[0] 1788747160.681109 read_OD 0x192000 thru 0x192fff (0x1000 bytes) 000482[1] 1788747160.687522 punch from 0x607e04 to 0x6102ed (0x84ea bytes) 000485[1] 1788747160.730845 trunc from 0x90b898 to 0x02404f (0x8e7848 bytes) 000486[0] 1788747160.744485 punch from 0x00913d to 0x018d07 (0xfbcb bytes) 000487[0] 1788747160.763689 trunc from 0x024050 to 0x39cb98 (0x378b49 bytes) 000490[0] 1788747160.818531 mapwrite 0x845000 thru 0x8478e7 (0x28e8 bytes) 000491[1] 1788747160.941896 trunc from 0x8478e8 to 0x404c0c (0x442cdb bytes) 000492[1] 1788747160.988625 read 0x2cc000 thru 0x2d0032 (0x4033 bytes) 000493[0] 1788747161.001482 mapread 0x3e0000 thru 0x3e6c4e (0x6c4f bytes) 000494[0] 1788747161.043654 trunc from 0x404c0d to 0x819b9f (0x414f93 bytes) 000495[0] 1788747161.078880 trunc from 0x819ba0 to 0x65dee1 (0x1bbcbe bytes) 000498[1] 1788747161.112530 write 0x85e000 thru 0x86ce50 (0xee51 bytes) 000499[1] 1788747161.137232 mapwrite 0x7eb000 thru 0x7ef8a3 (0x48a4 bytes) 000500[1] 1788747161.208954 mapwrite 0x462000 thru 0x4656bd (0x36be bytes) All operations completed A-OK! PASS 16a (48s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 22:12:45 (1788747165) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.383067 s, 21.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.310853 s, 27.0 MB/s Chance of close/open is 1 in 50 Seed set to 7169 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x88f097 000008[1] 1788747169.960134 trunc from 00000000 to 0x88f096 (0x88f097 bytes) 000009[1] 1788747170.032826 read 0x2830c2 thru 0x2848b0 (0x17ef bytes) 000010[1] 1788747170.066907 trunc from 0x88f097 to 0x2c3f24 (0x5cb172 bytes) 000012[0] 1788747170.115443 mapread 0x176c90 thru 0x1789a5 (0x1d16 bytes) 000013[1] 1788747170.161390 mapwrite 0x9027c8 thru 0x9047c2 (0x1ffb bytes) 000014[1] 1788747170.269303 trunc from 0x9047c3 to 0x85958d (0xab235 bytes) 000015[1] 1788747170.299599 punch from 0x82ecd9 to 0x83081f (0x1b47 bytes) 000017[0] 1788747170.331331 trunc from 0x85958e to 0x760b61 (0xf8a2c bytes) 000018[0] 1788747170.396518 trunc from 0x760b62 to 0x06028e (0x7008d3 bytes) 000019[1] 1788747170.452931 mapread 0x012167 thru 0x0128c2 (0x75c bytes) truncating to largest ever: 0x9ff155 000020[1] 1788747170.502017 trunc from 0x06028f to 0x9ff154 (0x99eec6 bytes) 000022[0] 1788747170.552680 mapread 0x2c819b thru 0x2c9702 (0x1568 bytes) 000024[1] 1788747170.606837 trunc from 0x9ff155 to 0x14f824 (0x8af930 bytes) 000025[1] 1788747170.644836 punch from 0x14371a to 0x1456a9 (0x1f90 bytes) 000026[1] 1788747170.651284 punch from 0x0e3252 to 0x0e3674 (0x423 bytes) 000027[1] 1788747170.656089 write 0x8a17cd thru 0x8a2db4 (0x15e8 bytes) 000028[1] 1788747170.661129 trunc from 0x8a2db5 to 0x372705 (0x5306af bytes) 000029[0] 1788747170.710405 read 0x29570f thru 0x296cda (0x15cc bytes) 000030[0] 1788747170.738933 mapread 0x01f599 thru 0x020df2 (0x185a bytes) 000031[1] 1788747170.788875 trunc from 0x372706 to 0x1108c2 (0x261e43 bytes) 000033[1] 1788747170.876897 trunc from 0x1108c3 to 0x381926 (0x271064 bytes) 000035[1] 1788747170.913050 punch from 0x01a847 to 0x01a8f0 (0xaa bytes) 000036[1] 1788747170.916436 mapwrite 0x0ba6cd thru 0x0bc176 (0x1aaa bytes) 000038[1] 1788747171.031726 read 0x25f636 thru 0x2615d5 (0x1fa0 bytes) 000039[0] 1788747171.135005 mapwrite 0x4046aa thru 0x4052c6 (0xc1d bytes) 000040[0] 1788747171.383122 mapwrite 0x0d7b58 thru 0x0d871b (0xbc4 bytes) 000041[1] 1788747171.484444 mapread 0x388602 thru 0x389a0d (0x140c bytes) 000043[0] 1788747171.562232 punch from 0x3b8b98 to 0x3b9f98 (0x1401 bytes) 000044[1] 1788747171.581122 read 0x23e23e thru 0x2401d4 (0x1f97 bytes) 000045[1] 1788747171.610295 mapread 0x21ee98 thru 0x220608 (0x1771 bytes) 000046[0] 1788747171.661254 mapread 0x19ebb1 thru 0x1a0748 (0x1b98 bytes) 000047[1] 1788747171.672910 write 0x045dc4 thru 0x04749c (0x16d9 bytes) 000049[1] 1788747171.761928 punch from 0x03a1f8 to 0x03a51a (0x323 bytes) 000050[1] 1788747171.793037 trunc from 0x4052c7 to 0x9dc0cd (0x5d6e07 bytes) 000051[0] 1788747171.850890 trunc from 0x9dc0ce to 0x44e258 (0x58de75 bytes) 000053[0] 1788747171.888914 trunc from 0x44e259 to 0x363cbb (0xea59d bytes) 000055[1] 1788747171.942757 mapwrite 0x6bbee2 thru 0x6bdc50 (0x1d6f bytes) 000056[0] 1788747172.110090 trunc from 0x6bdc51 to 0x09bcbd (0x621f93 bytes) 000057[0] 1788747172.155376 write 0x025fb8 thru 0x0278e5 (0x192e bytes) 000058[0] 1788747172.200972 write 0x02c9d3 thru 0x02d2ec (0x91a bytes) 000059[1] 1788747172.236468 mapwrite 0x2cad75 thru 0x2cb969 (0xbf5 bytes) 000060[1] 1788747172.363276 punch from 0x1d0f65 to 0x1d1dbb (0xe57 bytes) 000062[0] 1788747172.403083 write 0x31c2a3 thru 0x31e014 (0x1d72 bytes) 000063[1] 1788747172.447751 mapwrite 0x1168c5 thru 0x116c58 (0x394 bytes) 000064[1] 1788747172.554244 trunc from 0x31e015 to 0x04ac18 (0x2d33fc bytes) 000065[1] 1788747172.585674 mapread 0x029351 thru 0x029e97 (0xb47 bytes) 000066[0] 1788747172.594670 punch from 0x01e86f to 0x01ffda (0x176c bytes) 000069[0] 1788747172.659018 read 0x011f9c thru 0x0125b0 (0x615 bytes) 000070[0] 1788747172.681796 write 0x146324 thru 0x146bda (0x8b7 bytes) 000071[1] 1788747172.701393 write 0x683f6c thru 0x6856ac (0x1741 bytes) 000073[1] 1788747172.741263 mapread 0x00e9ee thru 0x0107ab (0x1dbe bytes) 000074[0] 1788747172.816655 write 0x5df586 thru 0x5e06b9 (0x1134 bytes) 000076[1] 1788747172.853358 mapwrite 0x329da5 thru 0x32b2fc (0x1558 bytes) 000077[1] 1788747173.066131 read 0x18e3aa thru 0x18e3ef (0x46 bytes) 000078[0] 1788747173.102399 punch from 0x3b59c7 to 0x3b69df (0x1019 bytes) 000080[0] 1788747173.151567 trunc from 0x6856ad to 0x96c9d5 (0x2e7329 bytes) 000081[0] 1788747173.194151 write 0x7e049e thru 0x7e0c85 (0x7e8 bytes) 000083[0] 1788747173.314640 trunc from 0x96c9d6 to 0x43d921 (0x52f0b4 bytes) 000087[0] 1788747173.367955 read 0x35f4cc thru 0x360b8a (0x16bf bytes) 000090[1] 1788747173.370100 mapread 0x070d8b thru 0x071a2b (0xca1 bytes) 000092[1] 1788747173.432815 mapwrite 0x6bfffa thru 0x6c101a (0x1021 bytes) 000093[0] 1788747173.559326 write 0x8f581a thru 0x8f5aed (0x2d4 bytes) 000095[1] 1788747173.654627 mapread 0x51866a thru 0x5192ea (0xc81 bytes) 000096[1] 1788747173.710375 mapwrite 0x439a77 thru 0x43a752 (0xcdc bytes) 000099[0] 1788747173.824241 punch from 0x47fc79 to 0x47fe8e (0x216 bytes) 000100[1] 1788747173.840595 read 0x2aa18a thru 0x2ab76f (0x15e6 bytes) 000101[1] 1788747173.898879 write 0x47b261 thru 0x47b4d9 (0x279 bytes) 000103[0] 1788747173.980496 trunc from 0x8f5aee to 0x69e09c (0x257a51 bytes) 000104[0] 1788747174.102276 read 0x10e7f4 thru 0x10ebcd (0x3da bytes) 000105[1] 1788747174.118466 read 0x06ea3f thru 0x06f8c3 (0xe85 bytes) 000106[0] 1788747174.163259 mapread 0x3a18e5 thru 0x3a3851 (0x1f6d bytes) 000107[0] 1788747174.214575 read 0x654d6d thru 0x6567b3 (0x1a47 bytes) 000111[1] 1788747174.251711 mapwrite 0x02c1d7 thru 0x02dbb2 (0x19dc bytes) 000113[0] 1788747174.425033 mapread 0x4e9677 thru 0x4eab7b (0x1505 bytes) 000114[1] 1788747174.477207 punch from 0x0ad232 to 0x0ae24a (0x1019 bytes) 000115[1] 1788747174.510021 mapread 0x1cc2e3 thru 0x1cdaca (0x17e8 bytes) 000116[1] 1788747174.561244 trunc from 0x69e09d to 0x586283 (0x117e19 bytes) 000117[1] 1788747174.603762 write 0x9dfcfa thru 0x9e1160 (0x1467 bytes) 000118[0] 1788747174.644963 mapread 0x907b3d thru 0x9085cd (0xa91 bytes) 000121[1] 1788747174.749635 read 0x3069c5 thru 0x30897e (0x1fba bytes) 000122[0] 1788747174.780769 write 0x443382 thru 0x443e02 (0xa81 bytes) 000123[1] 1788747174.813125 punch from 0x5e698a to 0x5e834c (0x19c3 bytes) 000124[0] 1788747174.848483 mapwrite 0x0d7852 thru 0x0d7c01 (0x3b0 bytes) 000127[0] 1788747174.979879 write 0x8245eb thru 0x82507d (0xa93 bytes) 000129[1] 1788747175.011222 read 0x220745 thru 0x2223e4 (0x1ca0 bytes) 000130[1] 1788747175.044428 mapread 0x3a8853 thru 0x3a9ee0 (0x168e bytes) 000131[1] 1788747175.054537 mapwrite 0x0d5a76 thru 0x0d6ac7 (0x1052 bytes) 000132[0] 1788747175.139757 punch from 0x2dd552 to 0x2decc2 (0x1771 bytes) 000133[1] 1788747175.173089 punch from 0x02c846 to 0x02d2ce (0xa89 bytes) 000134[0] 1788747175.223483 punch from 0x5d15b2 to 0x5d31b8 (0x1c07 bytes) 000136[1] 1788747175.270108 punch from 0x1ee2de to 0x1ef4ba (0x11dd bytes) 000139[0] 1788747175.322941 punch from 0x9a4425 to 0x9a455e (0x13a bytes) 000142[1] 1788747175.378008 read 0x3f3b9a thru 0x3f5739 (0x1ba0 bytes) 000145[1] 1788747175.457533 write 0x90091d thru 0x901302 (0x9e6 bytes) 000148[0] 1788747175.533622 mapread 0x3afd63 thru 0x3b19aa (0x1c48 bytes) 000149[1] 1788747175.575042 read 0x8134ef thru 0x81508f (0x1ba1 bytes) 000151[1] 1788747175.596433 mapwrite 0x091906 thru 0x0938dd (0x1fd8 bytes) 000153[1] 1788747175.712608 mapwrite 0x9e629c thru 0x9e7628 (0x138d bytes) 000154[1] 1788747175.814112 mapwrite 0x7dd560 thru 0x7de66a (0x110b bytes) 000155[0] 1788747175.912800 write 0x739ac7 thru 0x73a6de (0xc18 bytes) 000156[1] 1788747175.955182 read 0x722bce thru 0x72315f (0x592 bytes) 000159[1] 1788747176.030662 read 0x375f22 thru 0x3773e6 (0x14c5 bytes) 000161[0] 1788747176.048849 read 0x64abaa thru 0x64cb5a (0x1fb1 bytes) 000162[1] 1788747176.071447 mapread 0x34b6ab thru 0x34cc5c (0x15b2 bytes) 000163[1] 1788747176.082270 write 0x05e5ae thru 0x05f0b8 (0xb0b bytes) 000164[1] 1788747176.091204 punch from 0x2923dc to 0x293a52 (0x1677 bytes) 000166[1] 1788747176.122367 trunc from 0x9e7629 to 0x23e1f4 (0x7a9434 bytes) 000168[0] 1788747176.161096 mapread 0x0312ce thru 0x031a16 (0x749 bytes) 000169[0] 1788747176.200381 mapread 0x1a1c4d thru 0x1a32bb (0x166f bytes) 000170[0] 1788747176.246217 mapread 0x207071 thru 0x2085e9 (0x1579 bytes) 000173[0] 1788747176.295089 write 0x8703f4 thru 0x8707ad (0x3ba bytes) 000174[0] 1788747176.320896 trunc from 0x8707ae to 0x09ac71 (0x7d5b3c bytes) 000176[1] 1788747176.359748 mapwrite 0x69d2c5 thru 0x69f1c4 (0x1f00 bytes) 000177[0] 1788747176.462642 read 0x3d4ae1 thru 0x3d6984 (0x1ea4 bytes) 000180[0] 1788747176.565822 mapread 0x14c283 thru 0x14d85d (0x15db bytes) 000182[1] 1788747176.574729 write 0x52c481 thru 0x52c8c9 (0x449 bytes) 000184[0] 1788747176.597111 read 0x2a1b72 thru 0x2a2afe (0xf8d bytes) 000188[0] 1788747176.609829 write 0x3b7603 thru 0x3b78d1 (0x2cf bytes) 000189[1] 1788747176.645878 read 0x69c06c thru 0x69c11b (0xb0 bytes) 000192[1] 1788747176.690108 mapwrite 0x9269f8 thru 0x92861d (0x1c26 bytes) 000193[0] 1788747176.808391 mapwrite 0x6b4e15 thru 0x6b6b11 (0x1cfd bytes) 000195[1] 1788747176.931986 mapwrite 0x759f23 thru 0x75a171 (0x24f bytes) 000196[1] 1788747177.023609 mapwrite 0x08c76d thru 0x08cac7 (0x35b bytes) 000198[0] 1788747177.097959 read 0x5711a4 thru 0x572e7d (0x1cda bytes) 000199[0] 1788747177.125093 trunc from 0x92861e to 0x39c2e0 (0x58c33d bytes) 000200[1] 1788747177.175588 trunc from 0x39c2e1 to 0x61fd33 (0x283a53 bytes) 000201[1] 1788747177.227951 punch from 0x145e04 to 0x147aff (0x1cfc bytes) 000203[0] 1788747177.260604 mapread 0x1f9046 thru 0x1faaee (0x1aa9 bytes) 000204[0] 1788747177.313525 punch from 0x60b2df to 0x60bba5 (0x8c7 bytes) 000207[0] 1788747177.387921 mapwrite 0x683a6f thru 0x685a43 (0x1fd5 bytes) 000208[1] 1788747177.525533 trunc from 0x685a44 to 0x24522a (0x440819 bytes) 000210[1] 1788747177.586050 read 0x20a418 thru 0x20b57e (0x1167 bytes) 000212[0] 1788747177.658921 punch from 0x083309 to 0x0852f9 (0x1ff1 bytes) 000213[0] 1788747177.690909 trunc from 0x24522b to 0x9482e5 (0x7030bb bytes) 000216[1] 1788747177.787407 punch from 0x4fe0b8 to 0x4ff56f (0x14b8 bytes) 000217[1] 1788747177.812771 trunc from 0x9482e6 to 0x1452b6 (0x80302f bytes) 000219[1] 1788747177.919841 read 0x13a32b thru 0x13c035 (0x1d0b bytes) 000220[1] 1788747177.989298 mapwrite 0x1c9b62 thru 0x1ca3be (0x85d bytes) 000223[0] 1788747178.116766 mapwrite 0x187fe3 thru 0x1892f6 (0x1314 bytes) 000227[0] 1788747178.392854 trunc from 0x1ca3bf to 0x33d7f6 (0x173438 bytes) 000228[1] 1788747178.475534 mapwrite 0x63cbe4 thru 0x63d09f (0x4bc bytes) 000229[0] 1788747178.759955 punch from 0x0a9331 to 0x0a9470 (0x140 bytes) 000230[1] 1788747178.837098 mapread 0x0df66f thru 0x0e08b5 (0x1247 bytes) 000234[0] 1788747178.894777 read 0x083e55 thru 0x084a32 (0xbde bytes) 000235[1] 1788747178.901546 read 0x2827d9 thru 0x283230 (0xa58 bytes) 000236[0] 1788747178.921853 trunc from 0x63d0a0 to 0x209198 (0x433f07 bytes) 000237[0] 1788747179.010763 write 0x4145bd thru 0x415be9 (0x162d bytes) 000240[0] 1788747179.046119 trunc from 0x415bea to 0x587493 (0x1718aa bytes) 000242[1] 1788747179.130551 mapread 0x0b8003 thru 0x0b99a4 (0x19a2 bytes) 000243[0] 1788747179.187078 punch from 0x40ca01 to 0x40d638 (0xc38 bytes) 000244[1] 1788747179.199933 mapread 0x263ad0 thru 0x26571a (0x1c4b bytes) 000246[0] 1788747179.264821 read 0x3a6065 thru 0x3a7881 (0x181d bytes) 000249[0] 1788747179.326487 write 0x09d878 thru 0x09f875 (0x1ffe bytes) 000254[1] 1788747179.386469 write 0x07c944 thru 0x07cbb9 (0x276 bytes) 000255[1] 1788747179.432766 mapread 0x3eb4f1 thru 0x3ebf6a (0xa7a bytes) 000256[0] 1788747179.467952 read 0x15b00c thru 0x15cbb3 (0x1ba8 bytes) 000259[1] 1788747179.511362 punch from 0x05cea9 to 0x05e78d (0x18e5 bytes) 000262[0] 1788747179.565049 trunc from 0x587494 to 0x0cc18e (0x4bb305 bytes) 000263[1] 1788747179.614892 punch from 0x03ee9a to 0x03f103 (0x26a bytes) 000266[0] 1788747179.718053 mapwrite 0x1ee3e3 thru 0x1ef6a9 (0x12c7 bytes) 000267[1] 1788747179.824747 read 0x1e158e thru 0x1e286d (0x12e0 bytes) 000268[0] 1788747179.851792 read 0x145569 thru 0x1461ff (0xc97 bytes) 000269[1] 1788747179.947432 mapwrite 0x8e3770 thru 0x8e4d82 (0x1613 bytes) 000270[1] 1788747180.091754 trunc from 0x8e4d83 to 0x7ae94d (0x136435 bytes) 000272[0] 1788747180.158620 read 0x447926 thru 0x447d57 (0x432 bytes) 000274[0] 1788747180.217722 write 0x22e0d0 thru 0x22f54e (0x147f bytes) 000277[0] 1788747180.287553 punch from 0x586a9d to 0x58895e (0x1ec2 bytes) 000278[0] 1788747180.325917 read 0x5ac4f8 thru 0x5ad5f0 (0x10f9 bytes) 000280[0] 1788747180.344862 read 0x687b7f thru 0x68996a (0x1dec bytes) 000281[1] 1788747180.366862 punch from 0x15b4ba to 0x15d1ee (0x1d35 bytes) 000282[1] 1788747180.431146 mapwrite 0x343b80 thru 0x34446e (0x8ef bytes) 000284[1] 1788747180.582158 mapread 0x03e400 thru 0x03f907 (0x1508 bytes) 000285[1] 1788747180.642431 write 0x664509 thru 0x6648a1 (0x399 bytes) 000286[0] 1788747180.682928 write 0x2ee141 thru 0x2ee2a5 (0x165 bytes) 000287[1] 1788747180.719084 mapread 0x781b2a thru 0x782c32 (0x1109 bytes) 000289[1] 1788747180.781504 mapwrite 0x3f1b6e thru 0x3f20b2 (0x545 bytes) 000291[0] 1788747180.861771 read 0x3c9011 thru 0x3c95c4 (0x5b4 bytes) 000294[1] 1788747180.889909 mapwrite 0x8337a5 thru 0x833fb6 (0x812 bytes) 000299[1] 1788747180.972935 read 0x2c923b thru 0x2c9731 (0x4f7 bytes) 000300[0] 1788747180.976834 write 0x1dfe74 thru 0x1e04c4 (0x651 bytes) 000301[0] 1788747180.998573 mapwrite 0x23b26b thru 0x23bb02 (0x898 bytes) 000305[1] 1788747181.065897 punch from 0x606461 to 0x606c42 (0x7e2 bytes) 000306[1] 1788747181.084598 mapwrite 0x2fa6ee thru 0x2fbb7b (0x148e bytes) 000309[0] 1788747181.148101 mapwrite 0x457368 thru 0x4576d9 (0x372 bytes) 000311[1] 1788747181.254889 read 0x59f2ff thru 0x5a114a (0x1e4c bytes) 000313[0] 1788747181.291803 write 0x122572 thru 0x122d53 (0x7e2 bytes) 000314[1] 1788747181.328098 read 0x65286f thru 0x652bdb (0x36d bytes) 000317[0] 1788747181.352997 trunc from 0x833fb7 to 0x8fa52b (0xc6575 bytes) 000320[1] 1788747181.387748 mapwrite 0x3a417b thru 0x3a56b4 (0x153a bytes) 000321[0] 1788747181.476912 trunc from 0x8fa52c to 0x47bd86 (0x47e7a5 bytes) 000322[1] 1788747181.531362 trunc from 0x47bd87 to 0x03e09e (0x43dce8 bytes) 000323[1] 1788747181.570522 mapwrite 0x7d7140 thru 0x7d7980 (0x841 bytes) 000324[1] 1788747181.638222 mapwrite 0x26ab9c thru 0x26ae48 (0x2ad bytes) 000325[1] 1788747181.681863 mapread 0x7a6f16 thru 0x7a8e93 (0x1f7e bytes) 000326[0] 1788747181.728632 write 0x7d43cc thru 0x7d46e5 (0x31a bytes) 000329[1] 1788747181.780158 write 0x8ec337 thru 0x8edf6f (0x1c39 bytes) 000330[1] 1788747181.793478 mapwrite 0x47d421 thru 0x47f3b4 (0x1f94 bytes) 000331[0] 1788747181.891745 punch from 0x260b7b to 0x2619d7 (0xe5d bytes) 000332[0] 1788747181.923670 mapread 0x42880d thru 0x42a254 (0x1a48 bytes) 000333[1] 1788747181.958728 mapwrite 0x7736f8 thru 0x77548f (0x1d98 bytes) 000335[1] 1788747182.038688 mapread 0x64c5cd thru 0x64c6e6 (0x11a bytes) 000336[0] 1788747182.063629 mapwrite 0x1281b9 thru 0x128d45 (0xb8d bytes) 000337[0] 1788747182.124865 mapwrite 0x0292e5 thru 0x02b122 (0x1e3e bytes) 000339[0] 1788747182.261605 punch from 0x5a80d7 to 0x5a95a9 (0x14d3 bytes) 000340[1] 1788747182.281842 read 0x0415e9 thru 0x0435bf (0x1fd7 bytes) 000343[0] 1788747182.342432 mapwrite 0x92a434 thru 0x92b257 (0xe24 bytes) 000346[0] 1788747182.448844 mapread 0x1d0a09 thru 0x1d0aa5 (0x9d bytes) 000347[1] 1788747182.476903 mapread 0x82906d thru 0x8296bd (0x651 bytes) 000348[1] 1788747182.518300 read 0x601665 thru 0x60288e (0x122a bytes) 000349[1] 1788747182.540407 mapread 0x6ad7af thru 0x6af413 (0x1c65 bytes) 000351[0] 1788747182.558625 trunc from 0x92b258 to 0x03cca4 (0x8ee5b3 bytes) 000355[1] 1788747182.657607 mapwrite 0x30b798 thru 0x30bb7e (0x3e7 bytes) 000356[0] 1788747182.752011 read 0x1e5a39 thru 0x1e7406 (0x19ce bytes) 000357[0] 1788747182.829873 trunc from 0x30bb7f to 0x4b1143 (0x1a55c5 bytes) 000358[1] 1788747182.890051 write 0x92ba1c thru 0x92ccce (0x12b3 bytes) 000362[0] 1788747183.051834 punch from 0x661d45 to 0x662241 (0x4fd bytes) 000363[1] 1788747183.087743 write 0x73529c thru 0x735419 (0x17e bytes) 000365[1] 1788747183.119198 read 0x2ee85a thru 0x2efde4 (0x158b bytes) 000366[0] 1788747183.164134 mapwrite 0x316610 thru 0x31716f (0xb60 bytes) 000369[0] 1788747183.296728 punch from 0x20b917 to 0x20bfcf (0x6b9 bytes) 000370[0] 1788747183.317014 read 0x0ab3c4 thru 0x0abc97 (0x8d4 bytes) 000371[0] 1788747183.330760 trunc from 0x92cccf to 0x07d97f (0x8af34f bytes) 000372[1] 1788747183.371809 mapread 0x02960b thru 0x029a9a (0x490 bytes) 000374[1] 1788747183.428555 mapread 0x039ca0 thru 0x03a47b (0x7dc bytes) 000377[1] 1788747183.477717 trunc from 0x07d980 to 0x4b029e (0x43291f bytes) 000381[0] 1788747183.633008 write 0x9ecd97 thru 0x9ed3c1 (0x62b bytes) 000382[1] 1788747183.668360 read 0x62e62f thru 0x62f2d9 (0xcab bytes) 000383[1] 1788747183.705140 mapread 0x634d2b thru 0x634f78 (0x24e bytes) 000384[0] 1788747183.738677 write 0x7e9bd0 thru 0x7eb68e (0x1abf bytes) 000387[1] 1788747183.815867 trunc from 0x9ed3c2 to 0x103b60 (0x8e9861 bytes) 000388[1] 1788747183.899390 mapread 0x05b1d2 thru 0x05d18c (0x1fbb bytes) 000389[0] 1788747183.923688 write 0x00777c thru 0x008154 (0x9d9 bytes) 000392[0] 1788747184.069895 read 0x098f67 thru 0x09a296 (0x1330 bytes) 000393[0] 1788747184.137827 trunc from 0x103b61 to 0x1966e8 (0x92b88 bytes) 000394[0] 1788747184.210114 mapread 0x149791 thru 0x14a593 (0xe03 bytes) 000395[0] 1788747184.293713 write 0x88b5bc thru 0x88d2f8 (0x1d3d bytes) 000396[0] 1788747184.301106 read 0x2be2cc thru 0x2bf923 (0x1658 bytes) 000397[0] 1788747184.332496 punch from 0x4d14af to 0x4d24d9 (0x102b bytes) 000398[0] 1788747184.366519 mapwrite 0x55cb42 thru 0x55d426 (0x8e5 bytes) 000399[1] 1788747184.418580 mapwrite 0x17679d thru 0x177974 (0x11d8 bytes) 000400[1] 1788747184.525121 read 0x235432 thru 0x235b4e (0x71d bytes) 000403[0] 1788747184.599787 mapwrite 0x92f313 thru 0x92f6df (0x3cd bytes) 000404[1] 1788747184.727278 read 0x570734 thru 0x5721e7 (0x1ab4 bytes) 000405[1] 1788747184.767571 trunc from 0x92f6e0 to 0x6f138b (0x23e354 bytes) 000406[0] 1788747184.807298 mapread 0x6deeb9 thru 0x6e063d (0x1785 bytes) 000407[1] 1788747184.901412 mapwrite 0x01e223 thru 0x01eac6 (0x8a4 bytes) 000408[1] 1788747185.015968 mapread 0x0409ff thru 0x041474 (0xa76 bytes) 000410[0] 1788747185.047097 trunc from 0x6f138c to 0x53f02c (0x1b235f bytes) 000413[0] 1788747185.189723 trunc from 0x53f02d to 0x859796 (0x31a76a bytes) 000416[1] 1788747185.261843 mapread 0x7fb5bb thru 0x7fb8e9 (0x32f bytes) 000417[1] 1788747185.299116 write 0x5111bd thru 0x5123a1 (0x11e5 bytes) 000419[0] 1788747185.352739 write 0x7429b4 thru 0x7447f8 (0x1e45 bytes) 000420[0] 1788747185.371337 mapread 0x79be25 thru 0x79d42f (0x160b bytes) 000423[1] 1788747185.391922 punch from 0x3b5cf4 to 0x3b763c (0x1949 bytes) 000424[1] 1788747185.424449 mapread 0x771d5c thru 0x7729cf (0xc74 bytes) 000426[1] 1788747185.477806 mapwrite 0x994b5f thru 0x995525 (0x9c7 bytes) 000427[1] 1788747185.585118 mapwrite 0x3b93a3 thru 0x3b947c (0xda bytes) 000428[0] 1788747185.628070 trunc from 0x995526 to 0x539d85 (0x45b7a0 bytes) 000429[0] 1788747185.702636 trunc from 0x539d86 to 0x06b1b4 (0x4cebd1 bytes) 000430[1] 1788747185.750108 punch from 0x06aee5 to 0x06b1b4 (0x2d0 bytes) 000431[0] 1788747185.791087 mapwrite 0x674a0f thru 0x675f7c (0x156e bytes) 000432[0] 1788747185.895775 write 0x7eadfe thru 0x7eba9d (0xca0 bytes) 000434[0] 1788747185.903880 mapwrite 0x499488 thru 0x49a84d (0x13c6 bytes) 000435[1] 1788747186.017405 trunc from 0x7eba9e to 0x42ec74 (0x3bce29 bytes) 000436[1] 1788747186.130798 mapwrite 0x053fb0 thru 0x054b1c (0xb6d bytes) 000437[1] 1788747186.234838 punch from 0x1ae931 to 0x1af092 (0x762 bytes) 000438[1] 1788747186.275390 trunc from 0x42ec75 to 0x672be6 (0x243f72 bytes) 000439[0] 1788747186.302740 mapread 0x62417a thru 0x624502 (0x389 bytes) 000442[1] 1788747186.372107 trunc from 0x672be7 to 0x68ab86 (0x17fa0 bytes) 000443[1] 1788747186.424073 punch from 0x109f03 to 0x10a0b3 (0x1b1 bytes) 000444[0] 1788747186.450950 trunc from 0x68ab87 to 0x437b83 (0x253003 bytes) 000445[0] 1788747186.497426 trunc from 0x437b84 to 0x9b550d (0x57d98a bytes) 000448[1] 1788747186.559600 trunc from 0x9b550e to 0x8dbbca (0xd9943 bytes) 000450[1] 1788747186.642004 punch from 0x343625 to 0x3443f6 (0xdd2 bytes) 000455[0] 1788747186.756459 trunc from 0x8dbbcb to 0x965dce (0x8a204 bytes) 000456[0] 1788747186.830731 read 0x677689 thru 0x677b76 (0x4ee bytes) 000457[1] 1788747186.859274 write 0x519c4a thru 0x51aa07 (0xdbe bytes) 000459[0] 1788747186.912974 punch from 0x16cf2e to 0x16e37b (0x144e bytes) 000460[1] 1788747186.934848 mapread 0x69a6e3 thru 0x69b1b0 (0xace bytes) 000462[1] 1788747186.986427 mapread 0x29ef03 thru 0x29ef30 (0x2e bytes) 000463[1] 1788747187.030567 write 0x39bbd5 thru 0x39d909 (0x1d35 bytes) 000466[1] 1788747187.118667 punch from 0x0f17a2 to 0x0f24df (0xd3e bytes) 000467[0] 1788747187.150126 write 0x7fd00d thru 0x7fe7cd (0x17c1 bytes) 000469[0] 1788747187.223930 punch from 0x724368 to 0x72572e (0x13c7 bytes) 000470[0] 1788747187.246584 write 0x7b6719 thru 0x7b71a6 (0xa8e bytes) 000472[1] 1788747187.290783 mapwrite 0x483a1f thru 0x484d9d (0x137f bytes) 000473[0] 1788747187.368431 write 0x833c6b thru 0x834c33 (0xfc9 bytes) 000475[1] 1788747187.423082 trunc from 0x965dcf to 0x3acea3 (0x5b8f2b bytes) 000479[0] 1788747187.565651 mapread 0x35a91b thru 0x35c403 (0x1ae9 bytes) 000480[1] 1788747187.609497 punch from 0x3952f9 to 0x396efb (0x1c03 bytes) 000481[1] 1788747187.653711 mapwrite 0x486d8d thru 0x487acf (0xd43 bytes) 000482[0] 1788747187.787891 write 0x2630c3 thru 0x2635ff (0x53d bytes) 000483[1] 1788747187.825071 mapread 0x1cca62 thru 0x1cdd13 (0x12b2 bytes) 000484[0] 1788747187.875618 trunc from 0x487ad0 to 0x73d900 (0x2b5e31 bytes) 000486[1] 1788747187.917137 write 0x28d448 thru 0x28e4b8 (0x1071 bytes) 000487[1] 1788747187.957636 punch from 0x543b4b to 0x544617 (0xacd bytes) 000488[1] 1788747188.006656 punch from 0x10babe to 0x10bc7e (0x1c1 bytes) 000489[1] 1788747188.035698 mapwrite 0x1af6b4 thru 0x1b0931 (0x127e bytes) 000493[0] 1788747188.146218 read 0x18ea74 thru 0x18fad6 (0x1063 bytes) 000496[1] 1788747188.187863 mapwrite 0x060b17 thru 0x060fcd (0x4b7 bytes) 000498[1] 1788747188.298315 write 0x44d154 thru 0x44e4eb (0x1398 bytes) 000499[1] 1788747188.311823 mapwrite 0x292695 thru 0x293e32 (0x179e bytes) All operations completed A-OK! PASS 16b (27s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 22:13:12 (1788747192) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (3s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 22:13:16 (1788747196) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.437956 s, 19.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.276222 s, 30.4 MB/s ldlm.namespaces.MGC192.168.201.137@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8a8148c21000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8a814963b000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8a8148c21000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8a814963b000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8a8148c21000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8a814963b000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 15.8411 s, 6.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.40896 s, 16.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.78943 s, 21.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.67692 s, 12.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.394 s, 9.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.53221 s, 29.7 MB/s PASS 16d (70s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 22:14:25 (1788747265) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.3426 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238519 s, 4.4 MB/s PASS 16e (9s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 22:14:34 (1788747274) TIMEOUT PASS 16f (30s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 22:15:04 (1788747304) TIMEOUT PASS 16g (29s) == sanityn test 16h: mmap read after truncate file ======= 22:15:33 (1788747333) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (9s) == sanityn test 16i: read after truncate file ============ 22:15:42 (1788747342) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (9s) == sanityn test 16j: race dio with buffered i/o ========== 22:15:52 (1788747352) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.72229 s, 15.2 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.36595 s, 19.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.42481 s, 10.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.15669 s, 22.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.77265 s, 14.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.28489 s, 11.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.03715 s, 8.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.86519 s, 14.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.36652 s, 11.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.31898 s, 6.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.65596 s, 5.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.94553 s, 8.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.21809 s, 8.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.05353 s, 6.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.2823 s, 6.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.57766 s, 7.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.07776 s, 6.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.48829 s, 5.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.63749 s, 5.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (45s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 22:16:37 (1788747397) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Sun Sep 6 22:17:35 EDT 2026 + fsx 2 done: rc=0 Sun Sep 6 22:17:35 EDT 2026 All operations completed A-OK! All operations completed A-OK! + fsx 3 done: rc=0 Sun Sep 6 22:17:36 EDT 2026 + fsx 4 done: rc=0 Sun Sep 6 22:17:36 EDT 2026 + fsx 5 done: rc=0 Sun Sep 6 22:17:36 EDT 2026 PASS 16k (68s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 22:17:46 (1788747466) fail_loc=0x8000030a PASS 17 (14s) == sanityn test 18: mmap sanity check =========================================================================================== 22:18:00 (1788747480) mmap test1: basic mmap operation (PASS, 0.12762s) mmap test2: MAP_PRIVATE not write back (PASS, 0.14535s) mmap test3: concurrent mmap ops on two nodes (PASS, 25.297s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.6047s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.23568s) mmap test6: check mmap write/read content on two nodes (PASS, 0.50725s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.6288s) mmap test11: multi-thread shared mmap (PASS, 1.5831s) mmap test10: mtime not change for readonly mmap access (PASS, 1.148s) PASS 18 (45s) == sanityn test 19: test concurrent uncached read races ========================================================================= 22:18:44 (1788747524) oleg137-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 22:18:50 (1788747530) PASS 20 (9s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 22:18:59 (1788747539) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (8s) == sanityn test 23: others should see updated atime while another read============================================================== 22:19:07 (1788747547) now is 1788747551 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7664 new atime is 1788747612 PASS 23 (71s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 22:20:18 (1788747618) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 22:20:27 (1788747627) PASS 24b (8s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 22:20:34 (1788747634) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (8s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 22:20:42 (1788747642) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (4s) == sanityn test 26a: allow mtime to get older ============ 22:20:46 (1788747646) utime: good mknod times 1788747647 <= 1788747648 <= 1788747648 for /mnt/lustre/f26a utime: good mknod times 1788747647 <= 1788747648 <= 1788747648 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (9s) == sanityn test 26b: sync mtime between ost and mds ====== 22:20:55 (1788747655) PASS 26b (12s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 22:21:07 (1788747667) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7664 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-06 22:21:10.000000000 -0400 Birth: 2026-09-06 22:21:09.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-06 22:21:10.000000000 -0400 Birth: 2026-09-06 22:21:09.000000000 -0400 PASS 26c (12s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 22:21:21 (1788747681) 0 PASS 30 (10s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 22:21:31 (1788747691) fail_loc=0x314 PASS 31a (9s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 22:21:40 (1788747700) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg137-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.137@tcp:/lustre 7542784 38912 7498752 1% /mnt/lustre2 PASS 31b (26s) == sanityn test 31r: open-rename(replace) race =========== 22:22:06 (1788747726) -rw-r--r-- 1 root root 0 Sep 6 22:22 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (11s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 22:22:17 (1788747737) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43211 Terminated tail -f $tfile PASS 31s (10s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 22:22:27 (1788747747) PASS 31t (9s) == sanityn test 32b: lockless i/o ======================== 22:22:36 (1788747756) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (5s) == sanityn test 32c: contention detection ================ 22:22:41 (1788747761) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.73469 s, 19.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.04882 s, 25.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.61534 s, 14.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.06294 s, 12.9 MB/s contention_seconds 4, contention_events 16 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.78493 s, 13.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.24827 s, 12.3 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.48131 s, 21.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.60714 s, 20.1 MB/s PASS 32c (45s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 22:23:28 (1788747808) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 22:23:32 (1788747812) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 22:23:36 (1788747816) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 22:23:40 (1788747820) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (5s) == sanityn test 34: no lock timeout under IO ============= 22:23:45 (1788747825) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg137-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a8148c21000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a8148c21000.ost_server_uuid in FULL state after 0 sec oleg137-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a8148c21000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a8148c21000.ost_server_uuid in IDLE state after 0 sec oleg137-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a8148c21000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a8148c21000.ost_server_uuid in IDLE state after 0 sec oleg137-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a8148c21000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a8148c21000.ost_server_uuid in IDLE state after 0 sec oleg137-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a8148c21000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a8148c21000.ost_server_uuid in IDLE state after 0 sec oleg137-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a8148c21000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a8148c21000.ost_server_uuid in IDLE state after 0 sec PASS 34 (144s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 22:26:09 (1788747969) fail_loc=0x80000317 Race attempt 0 Wait for 50526 50547 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1855: 50526 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.201.137@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.201.137@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 - open/close 589 (time 1788747988.63 total 10.01 last 58.85) - open/close 1169 (time 1788747998.64 total 20.02 last 57.96) - open/close 1952 (time 1788748008.65 total 30.03 last 78.22) - open/close 2575 (time 1788748018.65 total 40.03 last 62.29) - open/close 3216 (time 1788748028.67 total 50.05 last 63.98) - open/close 3781 (time 1788748038.67 total 60.05 last 56.46) total: 4000 open/close in 67.79 seconds: 59.01 ops/second PASS 35 (80s) == sanityn test 36: handle ESTALE/open-unlink correctly == 22:27:29 (1788748049) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg137-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.09613 s, 47.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7664 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7435264) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.842289 s, 62.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7664 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7428096) after(7479296) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.19214 s, 44.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7664 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7479296) after_dd(7439360) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.00217 s, 52.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7664 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7497728) after_dd(7439360) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.636148 s, 82.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7664 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7497728) after_dd(7428096) after(7479296) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.759973 s, 69.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7664 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7479296) after_dd(7434240) after(7485440) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.525433 s, 99.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7664 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7485440) after_dd(7417856) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.29487 s, 40.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7664 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7497728) after_dd(7439360) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.684734 s, 76.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7664 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7497728) after_dd(7439360) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.814609 s, 64.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7664 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7497728) after_dd(7428096) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.04088 s, 50.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7664 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7497728) after_dd(7439360) after(7490560) PASS 36 (367s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 22:33:36 (1788748416) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7664 - create 1313 (time 1788748430.11 total 10.00 last 131.30) - create 2616 (time 1788748440.12 total 20.01 last 130.14) - create 4012 (time 1788748450.12 total 30.02 last 139.54) - create 5364 (time 1788748460.13 total 40.02 last 135.13) - create 6367 (time 1788748470.13 total 50.03 last 100.27) - create 7465 (time 1788748480.13 total 60.03 last 109.78) - create 8665 (time 1788748490.14 total 70.03 last 119.92) - create 9722 (time 1788748500.14 total 80.04 last 105.65) total: 10000 create in 81.90 seconds: 122.11 ops/second PASS 37 (94s) == sanityn test 39a: file mtime does not change after rename ========================================================== 22:35:10 (1788748510) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 22:35:18 (1788748518) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 22:35:26 (1788748526) repeat after cancel_lru_locks PASS 39c (8s) == sanityn test 39d: sync write should update mtime ====== 22:35:34 (1788748534) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 22:35:43 (1788748543) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (19s) == sanityn test 40c: pdirops: link and others ======================================================================== 22:36:03 (1788748563) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (18s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 22:36:20 (1788748580) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (19s) == sanityn test 40e: pdirops: rename and others ======================================================================== 22:36:39 (1788748599) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (19s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 22:36:58 (1788748618) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (14s) == sanityn test 41b: pdirops: create vs create ======================================================================== 22:37:12 (1788748632) open: File exists Conflict PASS 41b (16s) == sanityn test 41c: pdirops: create vs link ======================================================================== 22:37:28 (1788748648) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 22:37:42 (1788748662) Conflict PASS 41d (15s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 22:37:57 (1788748677) Conflict PASS 41e (15s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 22:38:12 (1788748692) Conflict PASS 41f (15s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 22:38:27 (1788748707) Conflict PASS 41g (15s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 22:38:42 (1788748722) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (16s) == sanityn test 41i: reint_open: create vs create ======== 22:38:58 (1788748738) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1007s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 22:55:46 (1788749746) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (14s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 22:56:00 (1788749760) open: File exists Conflict PASS 42b (13s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 22:56:13 (1788749773) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (14s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 22:56:27 (1788749787) Conflict PASS 42d (14s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 22:56:42 (1788749802) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (14s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 22:56:55 (1788749815) Conflict PASS 42f (16s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 22:57:11 (1788749831) Conflict PASS 42g (16s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 22:57:27 (1788749847) Conflict PASS 42h (16s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 22:57:43 (1788749863) PASS 43a (73s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 22:58:56 (1788749936) Conflict PASS 43b (13s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 22:59:10 (1788749950) Conflict PASS 43c (13s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 22:59:23 (1788749963) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (14s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 22:59:37 (1788749977) Conflict PASS 43e (12s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 22:59:49 (1788749989) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (13s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 23:00:02 (1788750002) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (13s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 23:00:15 (1788750015) Conflict PASS 43h (14s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 23:00:29 (1788750029) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 23:00:32 (1788750032) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (126s) == sanityn test 43k: unlink vs create ==================== 23:02:38 (1788750158) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1257s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 23:23:35 (1788751415) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Sun Sep 6 23:23:43 EDT 2026 PASS 44a (13s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 23:23:48 (1788751428) open: File exists Conflict PASS 44b (12s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 23:24:00 (1788751440) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (12s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 23:24:12 (1788751452) Conflict PASS 44d (12s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 23:24:24 (1788751464) Conflict PASS 44e (13s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 23:24:37 (1788751477) Conflict PASS 44f (13s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 23:24:50 (1788751490) Conflict PASS 44g (12s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 23:25:02 (1788751502) Conflict PASS 44h (15s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 23:25:17 (1788751517) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (3s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 23:25:20 (1788751520) PASS 45a (112s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 23:27:12 (1788751632) Conflict PASS 45b (13s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 23:27:25 (1788751645) Conflict PASS 45c (13s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 23:27:39 (1788751659) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (13s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 23:27:52 (1788751672) Conflict PASS 45e (14s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 23:28:06 (1788751686) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (14s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 23:28:20 (1788751700) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (14s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 23:28:34 (1788751714) Conflict PASS 45h (12s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 23:28:46 (1788751726) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 23:28:49 (1788751729) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1105s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 23:47:14 (1788752834) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (12s) == sanityn test 46b: pdirops: link vs create ======================================================================== 23:47:26 (1788752846) open: File exists Conflict PASS 46b (13s) == sanityn test 46c: pdirops: link vs link ======================================================================== 23:47:39 (1788752859) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (13s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 23:47:52 (1788752872) Conflict PASS 46d (12s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 23:48:04 (1788752884) Conflict PASS 46e (13s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 23:48:17 (1788752897) Conflict PASS 46f (14s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 23:48:31 (1788752911) Conflict PASS 46g (15s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 23:48:46 (1788752926) Conflict PASS 46h (13s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 23:48:59 (1788752939) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 23:49:02 (1788752942) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (3s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 23:49:05 (1788752945) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 23:49:08 (1788752948) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 23:49:11 (1788752951) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 23:49:15 (1788752955) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 23:49:18 (1788752958) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 23:49:21 (1788752961) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (2s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 23:49:24 (1788752964) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0999605 s, 102 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (12s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 23:49:36 (1788752976) 0+1 records in 0+1 records out 158 bytes copied, 0.0481541 s, 3.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218986) - No such process PASS 51a (8s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 23:49:45 (1788752985) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0111361 s, 92.0 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694588 s, 15.1 MB/s 3145728 PASS 51b (20s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 23:50:04 (1788753004) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0267632 s, 38.3 kB/s PASS 51c (12s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 23:50:16 (1788753016) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306784 s, 34.2 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (7s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 23:50:23 (1788753023) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 read lease(1) has applied. PASS 51e (9s) == sanityn test 54: rename locking ======================= 23:50:32 (1788753032) llite.lustre-ffff8a8148c21000.xattr_cache=1 llite.lustre-ffff8a814963b000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 23:50:34.000000000 -0400 Modify: 2026-09-06 23:50:34.000000000 -0400 Change: 2026-09-06 23:50:34.000000000 -0400 Birth: 2026-09-06 23:50:33.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 23:50:41.000000000 -0400 Modify: 2026-09-06 23:50:41.000000000 -0400 Change: 2026-09-06 23:50:41.000000000 -0400 Birth: 2026-09-06 23:50:40.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 23:50:47.000000000 -0400 Modify: 2026-09-06 23:50:47.000000000 -0400 Change: 2026-09-06 23:50:47.000000000 -0400 Birth: 2026-09-06 23:50:46.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 23:50:47.000000000 -0400 Modify: 2026-09-06 23:50:48.000000000 -0400 Change: 2026-09-06 23:50:48.000000000 -0400 Birth: 2026-09-06 23:50:46.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 23:50:53.000000000 -0400 Modify: 2026-09-06 23:50:54.000000000 -0400 Change: 2026-09-06 23:50:54.000000000 -0400 Birth: 2026-09-06 23:50:52.000000000 -0400 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505022 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 23:50:53.000000000 -0400 Modify: 2026-09-06 23:50:54.000000000 -0400 Change: 2026-09-06 23:50:54.000000000 -0400 Birth: 2026-09-06 23:50:52.000000000 -0400 PASS 54 (32s) == sanityn test 55a: rename vs unlink target dir ========= 23:51:04 (1788753064) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (12s) == sanityn test 55b: rename vs unlink source dir ========= 23:51:16 (1788753076) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (13s) == sanityn test 55c: rename vs unlink orphan target dir == 23:51:29 (1788753089) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (18s) == sanityn test 55d: rename file vs link ================= 23:51:47 (1788753107) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (14s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 23:52:01 (1788753121) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (2s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 23:52:04 (1788753124) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (18s) == sanityn test 55g: rename: race with trylock =========== 23:52:22 (1788753142) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (22s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 23:52:44 (1788753164) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788753166 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.484437 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (135s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 23:54:59 (1788753299) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788753303 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.20866 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (117s) == sanityn test 60: Verify data_version behaviour ======== 23:56:56 (1788753416) PASS 60 (11s) == sanityn test 70a: cd directory && rm directory ======== 23:57:07 (1788753427) PASS 70a (9s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 23:57:17 (1788753437) PASS 70b (9s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 23:57:25 (1788753445) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00439422 s, 14.9 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 23:57:28 (1788753448) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0223541 s, 2.9 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (4s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 23:57:32 (1788753452) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 23:57:35 (1788753455) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0226297 s, 2.9 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (3s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 23:57:38 (1788753458) llite.lustre-ffff8a8148c21000.xattr_cache=1 llite.lustre-ffff8a814963b000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (8s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 23:57:46 (1788753466) llite.lustre-ffff8a8148c21000.xattr_cache=1 llite.lustre-ffff8a814963b000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (10s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 23:57:56 (1788753476) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0151519 s, 67.6 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0104576 s, 97.9 kB/s 231791: get lock2 231795: get lock1 231791: done 231791: get lock1 231795: done 231795: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231795: cannot get lock2: Resource deadlock avoided 231795: exit rc=0 231791: done 231791: exit rc=0 PASS 74 (12s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 23:58:08 (1788753488) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0500992 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394214 s, 26.6 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232376 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (13s) == sanityn test 76: Verify MDT open_files listing ======== 23:58:21 (1788753501) opening files: ........ closing files: ........ PASS 76 (65s) == sanityn test 77a: check FIFO NRS policy =============== 23:59:26 (1788753566) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.216832 s, 77.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476496 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069297 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072001 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113247 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888515 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161639 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1076 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133897 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191421 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174978 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170031 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162056 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226418 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183655 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21824 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a8148c21000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8a8148c21000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230895 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245951 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608042 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613718 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943695 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561948 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642404 s, 16.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402635 s, 26.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784782 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576551 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823785 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860229 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098999 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971097 s, 10.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924579 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118662 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114119 s, 9.2 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg137-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg137-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (9s) == sanityn test 77b: check CRR-N NRS policy ============== 23:59:35 (1788753575) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.234496 s, 71.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561868 s, 18.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718967 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586922 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147505 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153853 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174504 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153614 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1575 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180468 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154226 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215461 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15622 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161054 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233577 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24618 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263934 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00886926 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116769 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158085 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987009 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147382 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190063 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116877 s, 89.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873059 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784413 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710721 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673809 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624195 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547358 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491255 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501494 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476122 s, 22.0 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.338323 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536014 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266209 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766605 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13801 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876604 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821003 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713283 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557371 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233048 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237808 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106235 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13607 s, 7.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a8148c21000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8a8148c21000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267267 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167006 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159739 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212994 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a8148c21000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357239 s, 29.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022686 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016419 s, 63.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017633 s, 59.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157724 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289836 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025361 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280491 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390404 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393201 s, 26.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814606 s, 12.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315409 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460025 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371149 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478544 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471803 s, 22.2 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg137-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg137-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (13s) == sanityn test 77c: check ORR NRS policy ================ 23:59:48 (1788753588) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.220069 s, 76.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207814 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354319 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453703 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433099 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119486 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708559 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125198 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134689 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110301 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11297 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143353 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170081 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190088 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162002 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111931 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147169 s, 7.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a8148c21000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8a8148c21000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453502 s, 23.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061663 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631748 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256596 s, 40.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542485 s, 19.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490079 s, 21.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490487 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328729 s, 31.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391706 s, 26.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370395 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461945 s, 22.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485688 s, 21.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530103 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533007 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568149 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393125 s, 26.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg137-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg137-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.294268 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257454 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642417 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986814 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767042 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604193 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953004 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128391 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107029 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205756 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308741 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540274 s, 19.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8a8148c21000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145401 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155149 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236639 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273556 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342472 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a8148c21000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319005 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00718298 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00867642 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584492 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00705815 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893818 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921568 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109565 s, 95.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072236 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740966 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943534 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055211 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469334 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443049 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145715 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642252 s, 16.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg137-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg137-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.334179 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286164 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273378 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393538 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510144 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679741 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558077 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620531 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843177 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311329 s, 33.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8a8148c21000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768569 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102952 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114931 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146753 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196489 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1388 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140929 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884261 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342548 s, 30.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420837 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498298 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427646 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340993 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391658 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11301 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540597 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542773 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547318 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621496 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612165 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076831 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775692 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821901 s, 12.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a8148c21000.lock_unused_count=4 removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg137-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg137-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (15s) == sanityn test 77d: check TRR nrs policy ================ 00:00:03 (1788753603) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.2553 s, 65.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664769 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071178 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138354 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130934 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155669 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09172 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186111 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246383 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231606 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214719 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248519 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217305 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212945 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221266 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240768 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333189 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a8148c21000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8a8148c21000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625499 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649621 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863106 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222922 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282295 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392274 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342895 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356722 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359556 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865659 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059367 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610476 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662992 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698814 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721444 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777717 s, 13.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.308749 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042744 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423484 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083917 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436247 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229563 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860179 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026565 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267755 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803378 s, 13.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179354 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150044 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121745 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107108 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211411 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222279 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168091 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865838 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989684 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112688 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100675 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845768 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081951 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929325 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106812 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808454 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119408 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108426 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103154 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116988 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811201 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101248 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129796 s, 8.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.432694 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465958 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067379 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366352 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028461 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117759 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869328 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196219 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177731 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126026 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173933 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222943 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127828 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136176 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230185 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195679 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282386 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a8148c21000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8a8148c21000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351109 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477828 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325044 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11266 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411263 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024629 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133125 s, 78.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131033 s, 80.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00395839 s, 265 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092969 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12613 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153904 s, 68.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100292 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00486949 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10052 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270557 s, 38.8 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg137-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg137-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (17s) == sanityn test 77e: check TBF NID nrs policy ============ 00:00:20 (1788753620) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.201.37@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.176605 s, 95.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338728 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462697 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546379 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101336 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149103 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140881 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145219 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128963 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12657 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166551 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215912 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171701 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213384 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16654 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170811 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221287 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a8148c21000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8a8148c21000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364194 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827378 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429534 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395523 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101459 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524642 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467042 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549293 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582545 s, 18.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043584 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629266 s, 16.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714443 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725496 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771826 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080417 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968799 s, 10.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg137-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg137-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.269424 s, 62.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585107 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530543 s, 19.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577854 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728073 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161334 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143188 s, 7.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163039 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852627 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136165 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106476 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156437 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110826 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144743 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119368 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149259 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154418 s, 6.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a8148c21000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8a8148c21000.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609864 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101499 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404474 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952013 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424457 s, 24.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509505 s, 20.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530119 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506584 s, 20.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543615 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545027 s, 19.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621705 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589568 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606673 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651063 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458904 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801121 s, 13.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg137-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg137-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.265687 s, 63.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309017 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039984 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176927 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129254 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128212 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163499 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13871 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19754 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217185 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200448 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233467 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225885 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185617 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231081 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266859 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a8148c21000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8a8148c21000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265967 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493568 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507239 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859357 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873175 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751404 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249741 s, 42.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00624074 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144089 s, 72.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117901 s, 88.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125609 s, 83.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589259 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587612 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110045 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744953 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00919584 s, 114 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a8148c21000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411184 s, 25.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg137-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg137-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.2322 s, 72.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269998 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264792 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380356 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028555 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914638 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252791 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118885 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907257 s, 11.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888204 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115474 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723617 s, 14.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a8148c21000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170785 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167386 s, 6.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19283 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2058 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289325 s, 3.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8a8148c21000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909406 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754206 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133979 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129571 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124847 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433962 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125019 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978505 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901038 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987792 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557961 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894653 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904843 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172634 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993515 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969937 s, 10.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (23s) == sanityn test 77f: check TBF JobID nrs policy ========== 00:00:43 (1788753643) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg137-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.28206 s, 59.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340157 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513615 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102454 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460461 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128253 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743385 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886087 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16399 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107077 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102059 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159138 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138165 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195999 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169968 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211995 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a8148c21000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8a8148c21000.lock_unused_count=6 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15642 s, 6.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a8148c21000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086292 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110949 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092429 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949341 s, 11.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180795 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183472 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866982 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027416 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145711 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835447 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085769 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120674 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747918 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685907 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106193 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733717 s, 14.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg137-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg137-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg137-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.218037 s, 76.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459072 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583892 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300336 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937935 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253262 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959014 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143267 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213117 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657342 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214947 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175581 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154894 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205638 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16144 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163614 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144668 s, 7.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194016 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511206 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241618 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294382 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529805 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641984 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037305 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662986 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973725 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505438 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533128 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560697 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631646 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744936 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774034 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815405 s, 12.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg137-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.410998 s, 40.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239403 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241115 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667523 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203083 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069518 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151012 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183889 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148238 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150557 s, 7.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8a8148c21000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26571 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180226 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247191 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179922 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190334 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194457 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207452 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877057 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114893 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616959 s, 17.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124117 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696801 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823276 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135033 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954003 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756911 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795716 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963746 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781538 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107969 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088324 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110541 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966534 s, 10.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg137-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.282121 s, 59.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642127 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618709 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20157 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224884 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224957 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212331 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313123 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297159 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163017 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251288 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365475 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36558 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240631 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263491 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364238 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a8148c21000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8a8148c21000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320955 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00833167 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126326 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00369443 s, 284 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140294 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229163 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106503 s, 98.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00262695 s, 399 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00244641 s, 429 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425656 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301049 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298028 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186765 s, 56.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227173 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357769 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651035 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163166 s, 6.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (25s) == sanityn test 77g: Change TBF type directly ============ 00:01:08 (1788753668) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (11s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 00:01:19 (1788753679) oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg137-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg137-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg137-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg137-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 PASS 77h (13s) == sanityn test 77i: Change rank of TBF rule ============= 00:01:32 (1788753692) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (22s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 00:01:54 (1788753714) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.546351 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604688 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594387 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550653 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920292 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104629 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230795 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244487 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122696 s, 8.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202404 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180539 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168592 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225932 s, 4.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284217 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281927 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195625 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182642 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754874 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053634 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514008 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883283 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448771 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700384 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486786 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781675 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790943 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762907 s, 13.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931388 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799829 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669747 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799195 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676395 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112656 s, 9.3 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg137-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg137-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.33386 s, 14.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6006 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (66s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 00:03:00 (1788753780) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg137-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg137-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg137-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg137-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg137-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.25601 s, 13.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781787 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856014 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868268 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503565 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711854 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658085 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137338 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103782 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996411 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166716 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176173 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116545 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207647 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221104 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123304 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981831 s, 10.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524482 s, 20.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409634 s, 25.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529756 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658847 s, 15.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933898 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068961 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532417 s, 19.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777397 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705142 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118458 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064915 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799927 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767314 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131467 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112039 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100314 s, 10.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9828 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9827 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg137-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg137-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg137-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg137-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg137-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27664 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439484 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420921 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722075 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102042 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122687 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118429 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609614 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132364 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202819 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140883 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116588 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214756 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176042 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160871 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17951 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261445 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747458 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872874 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877244 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853722 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092202 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939512 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994059 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098177 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1112 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116539 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859791 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11328 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12004 s, 8.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124806 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960227 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100523 s, 10.4 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg137-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg137-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9299 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.997 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (153s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 00:05:33 (1788753933) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg137-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg137-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg137-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg137-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg137-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.33418 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714152 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561425 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775195 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112556 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152308 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17586 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179119 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183189 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196909 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862208 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164429 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188982 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16585 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185584 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21206 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254571 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348229 s, 30.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016784 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818157 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460794 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237372 s, 44.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105378 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0027057 s, 388 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00615685 s, 170 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00989031 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115399 s, 90.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116846 s, 89.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00350287 s, 299 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141262 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07998 s, 13.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156962 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205692 s, 51.0 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0191 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0406 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg137-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg137-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg137-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg137-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg137-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27023 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127349 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112765 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478616 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134179 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170404 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132825 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119758 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241317 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262112 s, 4.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218792 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223855 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282769 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278901 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254674 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299125 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248576 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00235398 s, 445 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116532 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114519 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12141 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00361541 s, 290 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108016 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125503 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123765 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123387 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113803 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112354 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130879 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107174 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834366 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845832 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580985 s, 18.0 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0318 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9654 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (156s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 00:08:09 (1788754089) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg137-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.524435 s, 32.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366732 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570679 s, 18.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154125 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115823 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148789 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18687 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17372 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185848 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177726 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178218 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206444 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200709 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20241 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213003 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210764 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247923 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954417 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749709 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445048 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109333 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132676 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516124 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109584 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155972 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671496 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695722 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110253 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163319 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130683 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105578 s, 9.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154006 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115681 s, 9.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg137-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg137-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.62093 s, 12.2 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92337 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.37@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.37@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.486724 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723222 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735156 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101381 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154379 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148578 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17521 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213099 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193987 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188606 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223759 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254833 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228357 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28308 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315881 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292226 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294984 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117098 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174512 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126407 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948032 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106015 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998458 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10443 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104848 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104802 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901143 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13732 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105128 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134247 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10179 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17055 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168368 s, 6.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0188 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99844 s, 10.5 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.37@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg137-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.47329 s, 35.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452106 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402574 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664089 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142306 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136528 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174602 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143168 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199357 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149686 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142147 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22393 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200229 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211867 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230136 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157186 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192833 s, 5.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555703 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547093 s, 19.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101056 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390429 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395846 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682212 s, 15.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444881 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726151 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482574 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05092 s, 20.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585662 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663988 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925414 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070272 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738197 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922214 s, 11.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.0631 s, 13.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.51678 s, 19.0 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg137-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.549325 s, 30.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486243 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781256 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700767 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922781 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621793 s, 16.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149387 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176703 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716196 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192406 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125591 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179324 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284961 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944715 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110874 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215861 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229477 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883489 s, 11.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994562 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388449 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118225 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600557 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169014 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971248 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118108 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178422 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177946 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118362 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148215 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114587 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102334 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125907 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124737 s, 8.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.5469 s, 9.1 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0439 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.96052 s, 13.2 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0011 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg137-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.90194 s, 18.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605035 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086025 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833823 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709865 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154281 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118706 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171599 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179674 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11551 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13545 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188459 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207994 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213822 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19457 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277143 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29194 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049396 s, 21.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141993 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667811 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789165 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131694 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176988 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179196 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173944 s, 6.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102091 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209687 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112788 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110267 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188945 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139868 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147879 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146695 s, 7.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9825 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.67 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg137-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.574511 s, 29.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04255 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620759 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371859 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831593 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677977 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515581 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18118 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170551 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135397 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191023 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165564 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231733 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19713 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133127 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265788 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287982 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653382 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555923 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927678 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00444001 s, 236 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966212 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589551 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510827 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520459 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101406 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593193 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997086 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779303 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074229 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900164 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761283 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799111 s, 13.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg137-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg137-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.57701 s, 12.2 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.911 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg137-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.585598 s, 28.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537561 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101968 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601201 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829906 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759134 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533259 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616253 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082951 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224497 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059053 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102668 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344877 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356562 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272858 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321518 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387498 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658834 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830532 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330153 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385655 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411921 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457631 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992695 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103402 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759949 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502543 s, 20.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690695 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922142 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955093 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741792 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973053 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678669 s, 15.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.5434 s, 9.1 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96295 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.2201 s, 9.3 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93051 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (443s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 00:15:32 (1788754532) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg137-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.615246 s, 27.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512034 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885963 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554205 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572047 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365269 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651149 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23561 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162115 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264421 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184546 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937036 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265691 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921934 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177103 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225985 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245275 s, 4.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a8148c21000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8a8148c21000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230224 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942435 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708121 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430899 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639012 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553853 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048324 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131918 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599284 s, 17.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066119 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577619 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621002 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735696 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662711 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994254 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777083 s, 13.5 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.18966 s, 14.6 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90482 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (51s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 00:16:23 (1788754583) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.37@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.37@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.484355 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299492 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461516 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377941 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014956 s, 70.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398079 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173268 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175675 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169494 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171529 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162005 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136336 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162378 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252568 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26735 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238764 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283759 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a8148c21000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118071 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115411 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724717 s, 14.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120135 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744751 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809216 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747381 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840043 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150678 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894481 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154107 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145558 s, 7.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940944 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949097 s, 11.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a8148c21000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129592 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108242 s, 9.7 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.93373 s, 15.1 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91861 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (52s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 00:17:15 (1788754635) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.37@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg137-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.417198 s, 40.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488574 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521431 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659916 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16107 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187712 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147933 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167925 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151763 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119202 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167467 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201621 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160863 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128411 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156687 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145067 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179368 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483602 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522408 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537523 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381975 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383647 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673252 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461693 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495495 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050004 s, 21.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497076 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653182 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658934 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058686 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956179 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706256 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653666 s, 16.0 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg137-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg137-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.40445 s, 16.4 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.40055 s, 19.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (44s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 00:17:59 (1788754679) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg137-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.611267 s, 27.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049933 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418597 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687059 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935577 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10891 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149126 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152213 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165463 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179684 s, 5.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237186 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192744 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237308 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265894 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182193 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185553 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186996 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726071 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796401 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936025 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733085 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699196 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084201 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900653 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644718 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102324 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721465 s, 14.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723907 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781096 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831773 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115219 s, 9.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867939 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111301 s, 9.4 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6683 s, 9.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99325 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.46989 s, 14.0 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9092 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (95s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 00:19:35 (1788754775) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg137-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.03686 s, 16.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495437 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510567 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124324 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100072 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124728 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191408 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24679 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218896 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2222 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229369 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199734 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226304 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296805 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296415 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246244 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316125 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542012 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058322 s, 18.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870437 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783417 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646574 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737311 s, 14.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948281 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996169 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854125 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840386 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841729 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100295 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849044 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108266 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100871 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116999 s, 9.0 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8629 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9665 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (76s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 00:20:50 (1788754850) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg137-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.629222 s, 26.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316952 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360623 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072414 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431973 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751687 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903166 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668204 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108511 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961049 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101231 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139464 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172996 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183231 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244945 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153855 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228897 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313178 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368512 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430389 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045009 s, 23.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428722 s, 24.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431805 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453569 s, 23.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574673 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845567 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061784 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738122 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647753 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679334 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072491 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076615 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749803 s, 14.0 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg137-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg137-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74172 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0055 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg137-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.556975 s, 30.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244507 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516676 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513357 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455359 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134133 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125024 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139505 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139173 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140579 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187254 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154056 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157252 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227679 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212336 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284535 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253729 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056471 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865504 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827439 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633302 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797472 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111344 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851624 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933547 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127411 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986819 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940206 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136814 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977928 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100846 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101596 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119151 s, 8.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5164 s, 10.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9224 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2247 s, 10.3 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9041 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (143s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 00:23:13 (1788754993) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg137-client.virtnet /mnt/lustre (opts:) Stopping client oleg137-client.virtnet /mnt/lustre opts: Stopping clients: oleg137-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg137-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg137-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg137-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.37,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg137-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.37,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg137-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.37,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69725/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.37,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg137-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing load_modules_local oleg137-server: kmemleak disabled oleg137-server: Loading modules from /home/green/git/lustre-release/lustre oleg137-server: detected 4 online CPUs by sysfs oleg137-server: MODOPTS_LIBCFS= oleg137-server: Force libcfs to create 2 CPU partitions oleg137-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg137-server: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre Starting client oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre Started clients oleg137-client.virtnet: 192.168.201.137@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre2 Starting client oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre2 Started clients oleg137-client.virtnet: 192.168.201.137@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8a8142ceb800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8a81496f7000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a8142ceb800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a81496f7000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.679017 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822931 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158514 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276521 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26937 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378456 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268941 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a81496f7000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8a81496f7000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344305 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400299 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386608 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376127 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361012 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.445824 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.435941 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411282 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.484055 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425183 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a81496f7000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8a81496f7000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357722 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350333 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33276 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346581 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351805 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.435498 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359598 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370663 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343783 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354026 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376521 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.462021 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371831 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37408 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370046 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371314 s, 2.8 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.81537 s, 15.4 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.20172 s, 20.2 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.502012 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093707 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11848 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142383 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672479 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182575 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278138 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217376 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192789 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309456 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329749 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224701 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311174 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333558 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326969 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333929 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267745 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a81496f7000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8a81496f7000.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194322 s, 5.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168595 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205336 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152908 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175696 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156707 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157819 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162989 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16683 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176647 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199456 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189998 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192047 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198058 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205181 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219227 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a81496f7000.lock_unused_count=2 removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.72943 s, 13.6 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99516 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (218s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 00:26:52 (1788755212) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg137-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (18s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 00:27:09 (1788755229) On MGS 192.168.201.137, active = nodemap.active=1 On MGS 192.168.201.137, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.590771 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744124 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699602 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936453 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17933 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213226 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218414 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273109 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202439 s, 5.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219778 s, 4.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22965 s, 4.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311017 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336895 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301296 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300799 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337449 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a81496f7000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8a81496f7000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346678 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337947 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338211 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339883 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347082 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343305 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355458 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365463 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353588 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358781 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361309 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363095 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363015 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363537 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.442782 s, 2.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385459 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372157 s, 2.8 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.99703 s, 15.0 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.28916 s, 19.8 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.201.137, TBF.id = On MGS 192.168.201.137, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.567124 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812459 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878048 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13334 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12207 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337247 s, 3.1 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff8a81496f7000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8a81496f7000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309797 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35222 s, 3.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318804 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296514 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315211 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338615 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323892 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353621 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32721 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419217 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a81496f7000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8a81496f7000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.603721 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286761 s, 36.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00496731 s, 211 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109047 s, 96.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00394383 s, 266 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031601 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00500787 s, 209 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0052864 s, 198 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00285153 s, 368 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173894 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407684 s, 25.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00431337 s, 243 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242359 s, 43.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265815 s, 39.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016271 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163522 s, 64.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469485 s, 22.3 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.43864 s, 12.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.94203 s, 26.6 MB/s Read runtime is 5 s, speed is 20.000000 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg137-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg137-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 On MGS 192.168.201.137, active = nodemap.active=1 On MGS 192.168.201.137, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.890542 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081931 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945296 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696092 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115629 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288564 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267882 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176511 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318394 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399488 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385573 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273035 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34504 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387625 s, 2.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.466682 s, 2.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.395722 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426456 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a81496f7000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8a81496f7000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161024 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163586 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165297 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197846 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16831 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178758 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172108 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174818 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176553 s, 5.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177983 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249981 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181234 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230519 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187454 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189015 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226099 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a81496f7000.lock_unused_count=1 removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg137-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg137-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.61578 s, 12.2 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91818 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.201.137, TBF.id = On MGS 192.168.201.137, active = nodemap.active=0 PASS 77kj (180s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 00:30:09 (1788755409) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (11s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 00:30:20 (1788755420) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg137-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.2422 s, 323 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg137-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.1585 s, 261 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (58s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 00:31:18 (1788755478) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.542143 s, 30.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301709 s, 34.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463921 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621794 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830303 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115304 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113939 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109965 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127807 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226209 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2371 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149803 s, 7.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a81496f7000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8a81496f7000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257445 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199327 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227695 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218061 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209817 s, 5.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8a81496f7000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372177 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411943 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430359 s, 24.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595504 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082589 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479059 s, 21.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538535 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551107 s, 19.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577818 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581951 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896569 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580817 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630213 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628973 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632967 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714839 s, 14.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg137-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg137-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.65943 s, 15.7 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.20281 s, 20.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.416227 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397573 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430251 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121568 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114751 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132249 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144684 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152471 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161255 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189316 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177825 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182012 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199454 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179802 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207095 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197384 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200974 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626953 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537608 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692397 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658974 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671352 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680527 s, 15.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096536 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084327 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075375 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830193 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849621 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102068 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932989 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094913 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967975 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144925 s, 7.2 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg137-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg137-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.54949 s, 13.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.35822 s, 19.6 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (93s) == sanityn test 77o: Changing rank should not panic ====== 00:32:52 (1788755572) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (16s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 00:33:08 (1788755588) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (140s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 00:35:29 (1788755729) mds.MDS.mdt.nrs_policies=tbf oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg137-client: oleg137-server: ssh exited with exit code 36 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg137-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (35s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 00:36:05 (1788755765) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273793 kill md_thread xargs PASS 77r (56s) == sanityn test 77s: Check TBF LRU shrinker ============== 00:37:01 (1788755821) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (67s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 00:38:09 (1788755889) oleg137-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg137-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg137-client: oleg137-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (11s) == sanityn test 79: xattr: intent error ================== 00:38:21 (1788755901) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935687 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 00:38:23.000000000 -0400 Modify: 2026-09-07 00:38:23.000000000 -0400 Change: 2026-09-07 00:38:23.000000000 -0400 Birth: 2026-09-07 00:38:22.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (22s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 00:38:44 (1788755924) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (4s) == sanityn test 80b: Accessing directory during migration ========================================================== 00:38:48 (1788755928) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (5s) == sanityn test 81a: rename and stat under striped directory ========================================================== 00:38:53 (1788755933) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (5s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 00:38:59 (1788755939) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 00:39:03 (1788755943) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 00:39:07 (1788755947) - open/close 662 (time 1788755965.72 total 10.00 last 66.19) - open/close 1306 (time 1788755975.72 total 20.00 last 64.40) - open/close 1970 (time 1788755985.72 total 30.00 last 66.39) total: 2500 open/close in 38.09 seconds: 65.64 ops/second snapshot_time 1788756090.902059509 secs.nsecs start_time 1788755951.228250064 secs.nsecs elapsed_time 139.673809445 secs.nsecs open 2505 samples [usecs] 136 15601 3875980 9955503100 close 2504 samples [usecs] 150 17484 947717 1060752463 mknod 2501 samples [usecs] 583 11765 3355969 7502244013 unlink 2500 samples [usecs] 721 126027 5470696 66677176520 mkdir 2 samples [usecs] 991 7013 8004 50164250 rmdir 2 samples [usecs] 8414 10073 18487 172260725 rename 2500 samples [usecs] 451 70886 18934936 304578864808 getattr 3762 samples [usecs] 7 10626 243556 438877386 parallel_rename_file 2500 samples [usecs] 451 70886 18934936 304578864808 crossdir_rename 2500 samples [usecs] 451 70886 18934936 304578864808 readpage 25 samples [usecs] 3954 76048 627417 23263443229 PASS 81d (150s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 00:41:37 (1788756097) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7664 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7664 PASS 82 (9s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 00:41:46 (1788756106) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 00:41:50 (1788756110) fail_loc=0x8000060b PASS 84 (14s) == sanityn test 85: Lustre API root cache race =========== 00:42:04 (1788756124) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788756126 Finishing test test1 at 1788756131 PASS 85 (12s) == sanityn test 90: open/create and unlink striped directory ========================================================== 00:42:16 (1788756136) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (4s) == sanityn test 91: chmod and unlink striped directory === 00:42:20 (1788756140) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 00:42:23 (1788756143) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 00:42:27 (1788756147) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5786 0x169a 0x240000400 1 5810 0x16b2 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5787 0x169b 0x240000400 1 5811 0x16b3 0x280000400 rc1=0 and rc2=0 PASS 93 (21s) == sanityn test 94: signal vs CP callback race =========== 00:42:48 (1788756168) osc.lustre-OST0000-osc-ffff8a8142ceb800.idle_timeout=0 osc.lustre-OST0000-osc-ffff8a81496f7000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8a8142ceb800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8a81496f7000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.267706 s, 30.6 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.94745 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 290670 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0602702 s, 68.0 kB/s osc.lustre-OST0000-osc-ffff8a8142ceb800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8a81496f7000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a8142ceb800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a81496f7000.idle_timeout=debug PASS 94 (17s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 00:43:06 (1788756186) llite.lustre-ffff8a8142ceb800.fast_read=0 llite.lustre-ffff8a81496f7000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0878553 s, 23.9 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8a8142ceb800.fast_read=1 llite.lustre-ffff8a81496f7000.fast_read=1 PASS 95a (21s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 00:43:26 (1788756206) llite.lustre-ffff8a8142ceb800.fast_read=0 llite.lustre-ffff8a81496f7000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0240969 s, 510 kB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8a8142ceb800.fast_read=1 llite.lustre-ffff8a81496f7000.fast_read=1 PASS 95b (14s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 00:43:40 (1788756220) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 00:43:44 (1788756224) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00861552 s, 475 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (9s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 00:43:54 (1788756234) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.169497 s, 12.4 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (9s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 00:44:02 (1788756242) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.213263 s, 9.8 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (8s) == sanityn test 100e: DoM: read on open and file size ==== 00:44:10 (1788756250) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00649334 s, 1.8 MB/s PASS 100e (9s) == sanityn test 101a: Discard DoM data on unlink ========= 00:44:19 (1788756259) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0105424 s, 389 kB/s PASS 101a (9s) == sanityn test 101b: Discard DoM data on rename ========= 00:44:28 (1788756268) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0348685 s, 117 kB/s PASS 101b (9s) == sanityn test 101c: Discard DoM data on close-unlink === 00:44:37 (1788756277) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00555472 s, 737 kB/s PASS 101c (8s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 00:44:45 (1788756285) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 68 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf68:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (8s) == sanityn test 103: Test size correctness with lockahead ========================================================== 00:44:53 (1788756293) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788756296 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788756300 Finishing test test23 at 1788756300 PASS 103 (14s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 00:45:07 (1788756307) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 00:45:10 (1788756310) fail_loc=0x416 PASS 105 (24s) == sanityn test 106a: Verify the btime via statx() ======= 00:45:34 (1788756334) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 00:45:37 (1788756337) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0258041 s, 39.7 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938223 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 00:45:39.000000000 -0400 Modify: 2026-09-07 00:45:39.000000000 -0400 Change: 2026-09-07 00:45:39.000000000 -0400 Birth: 2026-09-07 00:45:38.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938223 -rw-r--r-- 1 0 0 1788756338 1788756339 1788756339 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938223 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 00:45:39.000000000 -0400 Modify: 2026-09-07 00:45:39.000000000 -0400 Change: 2026-09-07 00:45:39.000000000 -0400 Birth: 2026-09-07 00:45:38.000000000 -0400 1788756339 1024 1 PASS 106b (8s) == sanityn test 106c: Verify statx attributes mask ======= 00:45:45 (1788756345) PASS 106c (7s) == sanityn test 107a: Basic grouplock conflict =========== 00:45:53 (1788756353) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.383765 s, 27.3 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5796 0x16a4 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7664 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7664 First grouplock blocks second one PASS 107a (10s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 00:46:03 (1788756363) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.374931 s, 28.0 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5818 0x16ba 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7664 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (16s) == sanityn test 108a: lseek: parallel updates ============ 00:46:20 (1788756380) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00434148 s, 2.3 MB/s llite.lustre-ffff8a8142ceb800.hybrid_io=0 llite.lustre-ffff8a81496f7000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 5.03618 s, 1.7 MB/s PASS 108a (14s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 00:46:34 (1788756394) 192.168.201.137@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg137-client.virtnet /mnt/lustre (opts:) 192.168.201.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg137-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre2 Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre 192.168.201.137@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg137-client.virtnet /mnt/lustre (opts:) Stopping client oleg137-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing load_modules_local oleg137-server: kmemleak disabled oleg137-server: Loading modules from /home/green/git/lustre-release/lustre oleg137-server: detected 4 online CPUs by sysfs oleg137-server: MODOPTS_LIBCFS= oleg137-server: Force libcfs to create 2 CPU partitions oleg137-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre2 PASS 109 (49s) == sanityn test 110: do not grant another lock on resend ========================================================== 00:47:23 (1788756443) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 00:47:25.000000000 -0400 Modify: 2026-09-07 00:47:25.000000000 -0400 Change: 2026-09-07 00:47:25.000000000 -0400 Birth: 2026-09-07 00:47:25.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg137-client.virtnet oleg137-client.virtnet PASS 110 (72s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 00:48:35 (1788756515) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (5s) == sanityn test 112: update max-inherit in default LMV === 00:48:40 (1788756520) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 00:48:44 (1788756524) lustre-OST0000-osc-ffff8a816bcc1800 active. lustre-OST0001-osc-ffff8a816bcc1800 active. lustre-OST0000-osc-ffff8a816bcc1800 active. lustre-OST0001-osc-ffff8a816bcc1800 active. MGC192.168.201.137@tcp active. PASS 113 (9s) == sanityn test 114: implicit default LMV inherit ======== 00:48:53 (1788756533) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (4s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 00:48:57 (1788756537) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (4s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 00:49:01 (1788756541) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (4s) == sanityn test 121: trunc append race =================== 00:49:05 (1788756545) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0240836 s, 850 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.125619 s, 163 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (11s) == sanityn test 122: directory size is consistent across mounts ========================================================== 00:49:16 (1788756556) PASS 122 (11s) == sanityn test 200: service remains healthy while able to process request ========================================================== 00:49:27 (1788756567) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2065 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2634 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2506 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.342 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2251 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2332 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2975 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1535 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.276 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2753 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2969 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3042 s, 0.1 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3028 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.4796 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.467 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4562 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.5762 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.5712 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg137-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a8150d3d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a8150d3d800.ost_server_uuid in IDLE state after 0 sec PASS 200 (83s) cleanup: ====================================================== == sanityn test complete, duration 9963 sec ============== 00:50:52 (1788756652) === sanityn: start cleanup 00:50:54 (1788756654) === Stopping clients: oleg137-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg137-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 00:55:19 (1788756919) ===