-----============= acceptance-small: sanityn ============----- Mon Aug 31 13:54:18 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg458-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 13:54:42 (1788198882) === Starting client oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre2 Mount client oleg458-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre2 Started clients oleg458-client.virtnet: 192.168.204.158@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg458-client.virtnet: executing check_config_client /mnt/lustre oleg458-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg458-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff99d2c34e9000.idle_timeout=debug osc.lustre-OST0000-osc-ffff99d2d2ced800.idle_timeout=debug osc.lustre-OST0001-osc-ffff99d2c34e9000.idle_timeout=debug osc.lustre-OST0001-osc-ffff99d2d2ced800.idle_timeout=debug disable quota as required oleg458-server: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 13:55:27 (1788198927) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353981 s, 29.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7634] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 13:55:30 (1788198930) PASS 0 (10s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 13:55:39 (1788198939) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 13:55:47 (1788198947) -rw-r--r-- 1 root root 0 Aug 31 13:55 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 13:55:55 (1788198955) -rw-r--r-- 1 root root 0 Aug 31 13:55 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (7s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 13:56:03 (1788198963) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 13:56:09 (1788198969) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (9s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 13:56:18 (1788198978) total 1 -rwxrwxrwx 1 root root 0 Aug 31 13:55 f2a -rwxrwxrwx 1 root root 0 Aug 31 13:55 f2b total 1 -rwxrwxrwx 1 root root 0 Aug 31 13:55 f2a -rwxrwxrwx 1 root root 0 Aug 31 13:55 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0153891 s, 33.3 kB/s PASS 2e (9s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 13:56:27 (1788198987) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 13:56:31 (1788198991) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.659009 s, 3.2 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (9s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 13:56:40 (1788199000) PASS 3 (10s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 13:56:50 (1788199010) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 13:56:58 (1788199018) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (8s) == sanityn test 6: remove of open file on other node ============================================================================ 13:57:07 (1788199027) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (9s) == sanityn test 7: remove of open directory on other node ======================================================================= 13:57:17 (1788199037) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (8s) == sanityn test 8: remove of open special file on other node ==================================================================== 13:57:25 (1788199045) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (8s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 13:57:33 (1788199053) PASS 9a (8s) == sanityn test 9b: append to striped sparse file ======== 13:57:41 (1788199061) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100594 s, 10.4 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.109058 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (8s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 13:57:50 (1788199070) 1+0 records in 1+0 records out 1 byte copied, 0.016196 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0287864 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0236632 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.039272 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0181532 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0259505 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0208171 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0146539 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0170522 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0149346 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0301189 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0254415 s, 0.0 kB/s PASS 10a (10s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 13:58:00 (1788199080) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.029348 s, 105 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0640697 s, 63.9 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000477515 s, 6.4 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 13:58:08 (1788199088) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (9s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 13:58:16 (1788199096) warning: '-runtime' deprecated, use '-t runtime' instead - link 274 (time 1788199108.92 total 10.03 last 27.31) - link 582 (time 1788199118.94 total 20.06 last 30.73) - link 888 (time 1788199128.97 total 30.08 last 30.51) - link 1176 (time 1788199139.03 total 40.14 last 28.65) - link 1489 (time 1788199149.03 total 50.14 last 31.28) - link 1783 (time 1788199159.05 total 60.16 last 29.35) - link 2089 (time 1788199169.06 total 70.17 last 30.57) - link 2419 (time 1788199179.10 total 80.21 last 32.87) - link 2699 (time 1788199189.10 total 90.21 last 28.00) total: 2987 link in 100.04 seconds: 29.86 ops/second using seed 3045955684 running for 100 seconds - stat 4611 (time 1788199111 ; total 11 ; last 11) - stat 10000 (time 1788199121 ; total 21 ; last 10) - stat 15370 (time 1788199132 ; total 32 ; last 11) - stat 20000 (time 1788199142 ; total 42 ; last 10) - stat 24831 (time 1788199153 ; total 53 ; last 11) - stat 27666 (time 1788199164 ; total 64 ; last 11) - stat 30000 (time 1788199172 ; total 72 ; last 8) - stat 33226 (time 1788199183 ; total 83 ; last 11) - stat 35767 (time 1788199194 ; total 94 ; last 11) total: 37289 stats in 100 seconds: 372.890015 stats/second - unlinked 0 (time 1788199149 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2987) error: No such file or directory total: 2986 unlinks in 86 seconds: 34.720932 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21442) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21444) - No such process PASS 12 (145s) == sanityn test 13: test directory page revocation ======= 14:00:41 (1788199241) f13 f13 PASS 13 (12s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 14:00:53 (1788199253) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22631 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (8s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 14:01:01 (1788199261) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23224 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (8s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 14:01:09 (1788199269) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23802 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 14:01:17 (1788199277) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24385 Terminated $DIR1/$tdir/sleep 60 PASS 14c (9s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 14:01:26 (1788199286) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24968 Terminated $DIR1/$tdir/sleep 60 PASS 14d (11s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 14:01:37 (1788199297) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (61s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 14:02:38 (1788199358) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.16115 s, 7.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.802871 s, 10.4 MB/s Chance of close/open is 1 in 50 Seed set to 9366 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[1] 1788199367.376590 mapwrite 0x8bc671 thru 0x8c79a3 (0xb333 bytes) 000003[1] 1788199367.650245 mapread 0x582410 thru 0x58e26c (0xbe5d bytes) 000004[0] 1788199367.698882 mapwrite 0x351e3a thru 0x357c41 (0x5e08 bytes) 000006[1] 1788199367.915786 mapread 0x13b68b thru 0x14957e (0xdef4 bytes) 000008[1] 1788199368.012947 mapwrite 0x147a32 thru 0x14f0db (0x76aa bytes) 000009[0] 1788199368.287475 punch from 0x6d04b4 to 0x6d0cfa (0x847 bytes) 000010[0] 1788199368.385102 write 0x7103cf thru 0x713b71 (0x37a3 bytes) 000011[0] 1788199368.427695 punch from 0x3ae563 to 0x3b52e9 (0x6d87 bytes) 000012[1] 1788199368.525705 read 0x035a6b thru 0x037e5d (0x23f3 bytes) 000013[1] 1788199368.574378 mapwrite 0x1673a7 thru 0x176b01 (0xf75b bytes) 000014[0] 1788199368.766780 read 0x167b37 thru 0x16af7a (0x3444 bytes) 000018[0] 1788199368.895368 punch from 0x1bd70a to 0x1be775 (0x106c bytes) 000019[0] 1788199369.004841 mapwrite 0x768329 thru 0x775ff9 (0xdcd1 bytes) 000022[1] 1788199369.202920 read 0x7ff983 thru 0x801569 (0x1be7 bytes) 000026[0] 1788199369.282787 mapwrite 0x247433 thru 0x2557b0 (0xe37e bytes) truncating to largest ever: 0x192d11 000028[1] 1788199369.708826 trunc from 0x8c79a4 to 0x192d10 (0x734c93 bytes) truncating to largest ever: 0x6adb5f 000033[0] 1788199369.971473 trunc from 0x192d11 to 0x6adb5e (0x51ae4e bytes) 000037[0] 1788199370.125188 mapwrite 0x33ce52 thru 0x34ca5a (0xfc09 bytes) 000043[1] 1788199370.364380 mapread 0x1af06e thru 0x1b6f1d (0x7eb0 bytes) 000044[0] 1788199370.443865 trunc from 0x6adb5f to 0x630b0c (0x7d052 bytes) 000045[1] 1788199370.585334 punch from 0x43e798 to 0x446870 (0x80d9 bytes) 000046[0] 1788199370.647605 mapwrite 0x9cea76 thru 0x9d4d3d (0x62c8 bytes) 000047[1] 1788199370.898788 trunc from 0x9d4d3e to 0x1b8fda (0x81bd63 bytes) 000048[1] 1788199370.951681 read 0x0fa16e thru 0x105a82 (0xb915 bytes) 000051[1] 1788199371.215946 read 0x082dd4 thru 0x0863f0 (0x361d bytes) truncating to largest ever: 0x6b7ac3 000052[0] 1788199371.266996 trunc from 0x1b8fdb to 0x6b7ac2 (0x4feae8 bytes) 000053[0] 1788199371.522489 read 0x10cb82 thru 0x115d8b (0x920a bytes) 000055[1] 1788199371.584820 punch from 0x45e140 to 0x461dba (0x3c7b bytes) 000056[0] 1788199371.679875 write 0x94337f thru 0x943c4e (0x8d0 bytes) 000057[0] 1788199371.705708 read 0x8779b0 thru 0x87b4e4 (0x3b35 bytes) truncating to largest ever: 0x901e06 000061[1] 1788199371.821017 trunc from 0x943c4f to 0x901e05 (0x41e49 bytes) 000062[0] 1788199371.873346 read 0x68c365 thru 0x694024 (0x7cc0 bytes) 000064[0] 1788199371.962365 mapread 0x606cda thru 0x606f0b (0x232 bytes) 000066[1] 1788199372.007180 read 0x4d4f25 thru 0x4e38e9 (0xe9c5 bytes) 000068[0] 1788199372.032357 write 0x0df7b6 thru 0x0ee9d9 (0xf224 bytes) 000072[0] 1788199372.147136 mapread 0x0cbbd9 thru 0x0d9186 (0xd5ae bytes) 000076[0] 1788199372.216005 mapwrite 0x6dabd4 thru 0x6e1af7 (0x6f24 bytes) 000080[0] 1788199372.549515 mapread 0x2684b4 thru 0x27654e (0xe09b bytes) 000082[0] 1788199372.736220 punch from 0x26b4a4 to 0x274b7f (0x96dc bytes) 000083[1] 1788199372.746976 mapwrite 0x7fff51 thru 0x80e4b6 (0xe566 bytes) 000084[0] 1788199373.081959 read 0x6ff692 thru 0x70961d (0x9f8c bytes) 000088[1] 1788199373.249560 trunc from 0x901e06 to 0x3d5caf (0x52c156 bytes) 000089[1] 1788199373.473758 write 0x95275a thru 0x95ddf8 (0xb69f bytes) 000091[1] 1788199373.543724 read 0x47745c thru 0x483183 (0xbd28 bytes) 000092[0] 1788199373.563142 write 0x9a4f26 thru 0x9b49ba (0xfa95 bytes) 000094[0] 1788199373.681738 mapread 0x28026f thru 0x28ec82 (0xea14 bytes) 000099[1] 1788199373.961088 mapwrite 0x74f64e thru 0x750872 (0x1225 bytes) 000101[0] 1788199374.109936 read 0x22ee2d thru 0x23ea72 (0xfc46 bytes) 000104[1] 1788199374.219200 read 0x5328b7 thru 0x53c635 (0x9d7f bytes) 000106[1] 1788199374.299556 mapread 0x4dee06 thru 0x4e3840 (0x4a3b bytes) 000107[0] 1788199374.463581 mapwrite 0x2811e5 thru 0x284bf4 (0x3a10 bytes) 000108[1] 1788199374.733116 mapread 0x72b3a1 thru 0x739930 (0xe590 bytes) 000111[1] 1788199374.907580 read 0x02662d thru 0x02f3e6 (0x8dba bytes) 000114[0] 1788199375.008386 read 0x8aeba1 thru 0x8b669b (0x7afb bytes) 000115[1] 1788199375.020370 mapwrite 0x0410f8 thru 0x047f90 (0x6e99 bytes) 000116[1] 1788199375.277714 mapread 0x3b2328 thru 0x3b7743 (0x541c bytes) 000117[1] 1788199375.361705 write 0x584f16 thru 0x5853fe (0x4e9 bytes) 000118[0] 1788199375.389652 write 0x8a464a thru 0x8ae1dd (0x9b94 bytes) 000119[0] 1788199375.487344 mapread 0x33f327 thru 0x348017 (0x8cf1 bytes) 000120[1] 1788199375.634336 read 0x89a2a5 thru 0x8a5a49 (0xb7a5 bytes) 000121[1] 1788199375.762965 mapread 0x09dfa2 thru 0x09f759 (0x17b8 bytes) 000123[0] 1788199375.841393 read 0x59f969 thru 0x5a395e (0x3ff6 bytes) 000125[1] 1788199375.894140 write 0x91f42f thru 0x924da9 (0x597b bytes) 000127[0] 1788199376.005196 punch from 0x75a473 to 0x75f821 (0x53af bytes) 000129[0] 1788199376.127515 read 0x5cec96 thru 0x5d6ec3 (0x822e bytes) 000131[0] 1788199376.228795 write 0x08a198 thru 0x08dbe0 (0x3a49 bytes) 000133[0] 1788199376.412521 mapread 0x2c8c16 thru 0x2cc525 (0x3910 bytes) 000135[0] 1788199376.511312 mapwrite 0x9a6039 thru 0x9af92f (0x98f7 bytes) 000137[0] 1788199376.694183 trunc from 0x9b49bb to 0x5f010d (0x3c48ad bytes) 000139[0] 1788199376.753462 write 0x095e38 thru 0x0a2bd7 (0xcda0 bytes) 000140[0] 1788199376.776602 mapread 0x5e63ac thru 0x5ec912 (0x6567 bytes) 000141[0] 1788199376.827279 write 0x573a1f thru 0x5744ee (0xad0 bytes) 000143[1] 1788199376.949561 mapread 0x576878 thru 0x5778f4 (0x107d bytes) 000145[0] 1788199377.086021 trunc from 0x5f010e to 0x4ac6f4 (0x143a19 bytes) 000146[1] 1788199377.209589 trunc from 0x4ac6f5 to 0x230849 (0x27beab bytes) 000147[0] 1788199377.317976 mapwrite 0x8831ba thru 0x892e5a (0xfca1 bytes) 000148[0] 1788199377.652718 mapread 0x0882ea thru 0x091caa (0x99c1 bytes) 000149[1] 1788199377.775344 write 0x758aa3 thru 0x75fc0f (0x716d bytes) 000150[0] 1788199377.872556 mapread 0x563712 thru 0x565999 (0x2288 bytes) 000151[1] 1788199377.946139 read 0x2cb2b2 thru 0x2daf53 (0xfca2 bytes) 000153[1] 1788199378.006648 mapwrite 0x76a791 thru 0x76c680 (0x1ef0 bytes) 000154[1] 1788199378.203417 trunc from 0x892e5b to 0x7b7402 (0xdba58 bytes) 000156[1] 1788199378.324759 trunc from 0x7b7403 to 0x616818 (0x1a0bea bytes) 000158[1] 1788199378.400433 read 0x49276b thru 0x493326 (0xbbc bytes) 000159[1] 1788199378.406926 read 0x282e87 thru 0x290962 (0xdadc bytes) 000164[1] 1788199378.574953 punch from 0x60c2ff to 0x616818 (0xa51a bytes) 000165[0] 1788199378.703718 punch from 0x5701d9 to 0x57e268 (0xe090 bytes) 000166[1] 1788199378.777306 mapread 0x014bc6 thru 0x01bd5d (0x7198 bytes) 000167[0] 1788199378.835888 trunc from 0x616819 to 0x7615b0 (0x14ad98 bytes) 000169[1] 1788199378.979142 read 0x35e300 thru 0x366b35 (0x8836 bytes) 000171[0] 1788199379.094553 punch from 0x399a43 to 0x3a99e6 (0xffa4 bytes) 000172[1] 1788199379.135403 mapread 0x14c693 thru 0x157289 (0xabf7 bytes) 000173[1] 1788199379.252511 punch from 0x06494d to 0x072560 (0xdc14 bytes) 000174[1] 1788199379.311038 mapwrite 0x0d47ed thru 0x0d4ccd (0x4e1 bytes) 000175[0] 1788199379.419001 mapwrite 0x1d7bd0 thru 0x1e513b (0xd56c bytes) 000179[1] 1788199379.782736 write 0x856b37 thru 0x858fbb (0x2485 bytes) 000180[1] 1788199379.854478 read 0x29d2d4 thru 0x2ac30a (0xf037 bytes) 000182[0] 1788199379.938856 trunc from 0x858fbc to 0x18c935 (0x6cc686 bytes) 000183[1] 1788199380.045125 punch from 0x00ee7f to 0x01ab2f (0xbcb1 bytes) 000184[1] 1788199380.130903 punch from 0x0f5556 to 0x0f9dfc (0x48a7 bytes) 000185[0] 1788199380.198508 mapwrite 0x3121df thru 0x314ed9 (0x2cfb bytes) 000186[0] 1788199380.468431 trunc from 0x314eda to 0x0f6ddc (0x21e0fd bytes) 000187[1] 1788199380.506351 write 0x40310f thru 0x40a1fe (0x70f0 bytes) 000188[1] 1788199380.541842 mapwrite 0x750eb3 thru 0x75d1dd (0xc32b bytes) 000189[0] 1788199380.724523 trunc from 0x75d1de to 0x0a2af7 (0x6ba6e6 bytes) 000190[1] 1788199380.790080 punch from 0x01ec86 to 0x02caf8 (0xde73 bytes) 000192[0] 1788199380.848714 write 0x493402 thru 0x4a236d (0xef6c bytes) 000193[0] 1788199380.886144 read 0x437b2d thru 0x4464ac (0xe980 bytes) 000195[0] 1788199380.938939 mapwrite 0x11cba5 thru 0x123277 (0x66d3 bytes) 000196[0] 1788199381.238600 punch from 0x33dd08 to 0x345670 (0x7969 bytes) 000198[0] 1788199381.243596 punch from 0x0c3f93 to 0x0d00fc (0xc16a bytes) truncating to largest ever: 0x902ce2 000204[1] 1788199381.308423 trunc from 0x4a236e to 0x902ce1 (0x460974 bytes) 000206[1] 1788199381.396297 write 0x9c2181 thru 0x9c3890 (0x1710 bytes) 000207[1] 1788199381.416030 write 0x978290 thru 0x983347 (0xb0b8 bytes) 000208[1] 1788199381.434184 trunc from 0x9c3891 to 0x6a369f (0x3201f1 bytes) 000209[0] 1788199381.461536 write 0x5d6ebd thru 0x5e31d5 (0xc319 bytes) 000210[1] 1788199381.506465 read 0x6a2f36 thru 0x6a369f (0x76a bytes) 000211[0] 1788199381.562033 mapread 0x5f15ed thru 0x5f9b94 (0x85a8 bytes) 000212[0] 1788199381.599494 write 0x361faa thru 0x36ea7e (0xcad5 bytes) 000213[0] 1788199381.666345 read 0x12cea5 thru 0x1357e5 (0x8941 bytes) 000214[1] 1788199381.688174 punch from 0x44668b to 0x44c4bd (0x5e33 bytes) 000215[1] 1788199381.750387 read 0x0b25d6 thru 0x0c123e (0xec69 bytes) 000217[1] 1788199381.837995 write 0x18daf3 thru 0x190837 (0x2d45 bytes) 000218[0] 1788199381.906789 mapwrite 0x7d3d85 thru 0x7d5fbf (0x223b bytes) truncating to largest ever: 0x920a93 000219[0] 1788199382.039527 trunc from 0x7d5fc0 to 0x920a92 (0x14aad3 bytes) 000220[1] 1788199382.060329 mapwrite 0x49a6ef thru 0x49e351 (0x3c63 bytes) 000221[1] 1788199382.159827 read 0x710cea thru 0x7201fe (0xf515 bytes) 000222[0] 1788199382.178285 punch from 0x16294e to 0x169371 (0x6a24 bytes) 000223[0] 1788199382.195516 mapwrite 0x056ae8 thru 0x05a37a (0x3893 bytes) 000224[1] 1788199382.275752 trunc from 0x920a93 to 0x1a1856 (0x77f23c bytes) 000226[0] 1788199382.333839 mapwrite 0x866d8e thru 0x873704 (0xc977 bytes) 000227[1] 1788199382.476383 read 0x0cd71c thru 0x0d3de2 (0x66c7 bytes) 000233[0] 1788199382.547492 mapread 0x7d3c26 thru 0x7db087 (0x7462 bytes) 000234[1] 1788199382.573894 trunc from 0x873705 to 0x89e3ce (0x2acca bytes) 000235[1] 1788199382.626199 write 0x30a15f thru 0x312380 (0x8222 bytes) 000236[0] 1788199382.655807 trunc from 0x89e3cf to 0x194f45 (0x709489 bytes) 000237[1] 1788199382.741601 punch from 0x04a6d3 to 0x055ccb (0xb5f9 bytes) 000238[1] 1788199382.804464 write 0x387807 thru 0x390ea7 (0x96a1 bytes) 000241[0] 1788199382.856167 write 0x7cbf69 thru 0x7d2709 (0x67a1 bytes) 000242[0] 1788199382.877065 mapwrite 0x6d385d thru 0x6dbe27 (0x85cb bytes) 000243[0] 1788199382.977764 punch from 0x28b16f to 0x292056 (0x6ee8 bytes) 000245[0] 1788199383.010459 write 0x98af93 thru 0x996711 (0xb77f bytes) 000247[0] 1788199383.046073 trunc from 0x996712 to 0x44b274 (0x54b49d bytes) 000248[1] 1788199383.096869 punch from 0x3481f9 to 0x35468d (0xc495 bytes) 000249[1] 1788199383.125560 read 0x357b70 thru 0x361835 (0x9cc6 bytes) 000250[0] 1788199383.153782 write 0x9ed55d thru 0x9f1b58 (0x45fc bytes) 000252[0] 1788199383.201590 mapwrite 0x90f25c thru 0x91c3ad (0xd152 bytes) 000253[0] 1788199383.434444 trunc from 0x9f1b59 to 0x00b0f9 (0x9e6a5f bytes) 000254[0] 1788199383.471708 write 0x3e921a thru 0x3ee372 (0x5159 bytes) 000255[1] 1788199383.551069 punch from 0x181bba to 0x1906ac (0xeaf3 bytes) 000256[1] 1788199383.643957 mapwrite 0x8170ab thru 0x820e0a (0x9d60 bytes) 000258[1] 1788199384.158478 mapwrite 0x5f6591 thru 0x605ef5 (0xf965 bytes) 000263[1] 1788199384.419005 mapread 0x5b13b2 thru 0x5b3801 (0x2450 bytes) 000266[0] 1788199384.495303 trunc from 0x820e0b to 0x06e134 (0x7b2cd6 bytes) 000267[1] 1788199384.560838 write 0x66f61d thru 0x674a43 (0x5427 bytes) 000268[1] 1788199384.592355 punch from 0x1fb1ec to 0x2095df (0xe3f4 bytes) 000272[1] 1788199384.641234 trunc from 0x674a44 to 0x6be48b (0x49a48 bytes) 000273[0] 1788199384.679597 write 0x171bcd thru 0x17d858 (0xbc8c bytes) 000274[0] 1788199384.709742 write 0x6f7d7d thru 0x6f8e3b (0x10bf bytes) 000275[0] 1788199384.728546 punch from 0x3af0e8 to 0x3b6cc2 (0x7bdb bytes) 000276[1] 1788199384.734170 trunc from 0x6f8e3c to 0x1b3dd5 (0x545066 bytes) 000277[1] 1788199384.805395 write 0x0b3b84 thru 0x0bd44a (0x98c7 bytes) 000278[0] 1788199384.839534 trunc from 0x1b3dd6 to 0x35d441 (0x1a966c bytes) 000279[0] 1788199384.936245 write 0x399257 thru 0x399934 (0x6de bytes) 000283[1] 1788199385.095581 mapwrite 0x4aa69f thru 0x4af046 (0x49a8 bytes) 000284[1] 1788199385.375733 trunc from 0x4af047 to 0x250308 (0x25ed3e bytes) 000285[0] 1788199385.490755 trunc from 0x250309 to 0x59e6a9 (0x34e3a1 bytes) 000287[0] 1788199385.677546 mapread 0x522635 thru 0x531b16 (0xf4e2 bytes) 000290[0] 1788199385.746730 mapwrite 0x80823a thru 0x8114cc (0x9293 bytes) 000292[0] 1788199385.939164 write 0x6d4d01 thru 0x6e3874 (0xeb74 bytes) 000293[0] 1788199385.955087 write 0x5c0f74 thru 0x5c71dc (0x6269 bytes) 000294[0] 1788199386.013114 read 0x7fa730 thru 0x805d93 (0xb664 bytes) 000296[0] 1788199386.032570 mapread 0x59a91c thru 0x5a028c (0x5971 bytes) 000298[0] 1788199386.098893 trunc from 0x8114cd to 0x68e0c4 (0x183408 bytes) 000299[0] 1788199386.164752 mapwrite 0x13e9a3 thru 0x14aaa4 (0xc102 bytes) 000300[1] 1788199386.419533 read 0x599329 thru 0x59e733 (0x540b bytes) 000301[1] 1788199386.511273 trunc from 0x68e0c5 to 0x2527f8 (0x43b8cc bytes) 000304[1] 1788199386.754134 mapread 0x20b61e thru 0x20dfbb (0x299e bytes) 000306[0] 1788199386.870251 write 0x2094cd thru 0x2134b1 (0x9fe5 bytes) 000307[1] 1788199386.972460 read 0x00ec90 thru 0x01b82d (0xcb9e bytes) 000308[0] 1788199387.150421 mapwrite 0x1a6046 thru 0x1a69f0 (0x9ab bytes) 000310[1] 1788199387.380582 read 0x0b3851 thru 0x0bcbf0 (0x93a0 bytes) 000313[1] 1788199387.517908 trunc from 0x2527f9 to 0x6e81fa (0x495a02 bytes) 000314[1] 1788199387.577337 trunc from 0x6e81fb to 0x17b61b (0x56cbdf bytes) 000316[1] 1788199387.691590 mapwrite 0x5694ff thru 0x572f77 (0x9a79 bytes) 000317[0] 1788199388.123195 trunc from 0x572f78 to 0x3d621f (0x19cd58 bytes) 000318[1] 1788199388.213467 mapwrite 0x91c3cf thru 0x9204fb (0x412d bytes) 000319[0] 1788199388.463940 punch from 0x1ac8b7 to 0x1ba0e1 (0xd82b bytes) 000320[0] 1788199388.503296 punch from 0x550ffc to 0x5537b5 (0x27ba bytes) 000321[1] 1788199388.565356 write 0x19fd74 thru 0x1ace50 (0xd0dd bytes) 000322[1] 1788199388.656498 mapwrite 0x28e0b7 thru 0x297acb (0x9a15 bytes) 000323[1] 1788199388.956848 read 0x7c3dd4 thru 0x7c6c7c (0x2ea9 bytes) 000326[1] 1788199389.028461 write 0x670d3f thru 0x6776d0 (0x6992 bytes) 000327[1] 1788199389.075168 mapwrite 0x0a2adc thru 0x0af992 (0xceb7 bytes) 000328[0] 1788199389.329430 write 0x2ac78a thru 0x2b9131 (0xc9a8 bytes) 000329[0] 1788199389.445284 write 0x139f22 thru 0x13ee71 (0x4f50 bytes) 000330[1] 1788199389.515695 punch from 0x35a307 to 0x35f228 (0x4f22 bytes) 000331[1] 1788199389.583084 read 0x082275 thru 0x08a4f7 (0x8283 bytes) 000332[1] 1788199389.628285 punch from 0x692623 to 0x6977ee (0x51cc bytes) 000333[1] 1788199389.650497 punch from 0x4a1124 to 0x4afa4a (0xe927 bytes) 000334[0] 1788199389.673565 mapwrite 0x877f3a thru 0x87c4c8 (0x458f bytes) 000335[1] 1788199389.860082 mapread 0x042d0b thru 0x04fd01 (0xcff7 bytes) 000336[0] 1788199389.931291 mapwrite 0x2454bb thru 0x24bd1a (0x6860 bytes) 000337[1] 1788199390.049380 read 0x664b38 thru 0x669015 (0x44de bytes) 000338[1] 1788199390.090734 mapwrite 0x50fb6f thru 0x516937 (0x6dc9 bytes) 000339[1] 1788199390.259994 punch from 0x75980f to 0x75b242 (0x1a34 bytes) 000340[0] 1788199390.282759 mapread 0x465e66 thru 0x46d639 (0x77d4 bytes) 000341[0] 1788199390.417688 read 0x86fa6c thru 0x87e905 (0xee9a bytes) 000343[0] 1788199390.503115 punch from 0x1b710b to 0x1bfd1d (0x8c13 bytes) 000345[1] 1788199390.570982 write 0x110f9e thru 0x11aaf6 (0x9b59 bytes) 000347[1] 1788199390.622193 mapwrite 0x3e7a7c thru 0x3f632c (0xe8b1 bytes) 000348[1] 1788199390.938778 read 0x1aca7a thru 0x1b9a32 (0xcfb9 bytes) 000350[0] 1788199390.997308 write 0x6c8342 thru 0x6d6fab (0xec6a bytes) 000351[0] 1788199391.094477 read 0x8810fb thru 0x88948b (0x8391 bytes) 000353[0] 1788199391.129465 mapread 0x3a50ef thru 0x3ad610 (0x8522 bytes) 000356[1] 1788199391.360166 write 0x4ede59 thru 0x4f33fa (0x55a2 bytes) 000357[1] 1788199391.455839 trunc from 0x9204fc to 0x589dcc (0x39672f bytes) 000358[0] 1788199391.496368 mapread 0x07fec9 thru 0x085cbe (0x5df6 bytes) 000359[1] 1788199391.583270 mapread 0x048558 thru 0x05332b (0xadd4 bytes) 000360[0] 1788199391.611196 trunc from 0x589dcd to 0x6c52fd (0x13b531 bytes) 000362[0] 1788199391.667178 read 0x1ed57d thru 0x1ee21b (0xc9f bytes) 000365[1] 1788199391.713127 trunc from 0x6c52fe to 0x881963 (0x1bc666 bytes) 000366[0] 1788199391.775753 write 0x44b35d thru 0x4523b6 (0x705a bytes) 000367[0] 1788199391.832579 punch from 0x4979e0 to 0x49a2dc (0x28fd bytes) 000369[0] 1788199391.896602 read 0x3d771e thru 0x3e5035 (0xd918 bytes) 000370[0] 1788199391.953605 punch from 0x848a24 to 0x856822 (0xddff bytes) 000372[0] 1788199392.003122 read 0x3dbeb8 thru 0x3e0193 (0x42dc bytes) 000374[0] 1788199392.045890 mapread 0x430d16 thru 0x436c52 (0x5f3d bytes) 000378[0] 1788199392.182586 mapwrite 0x2478ca thru 0x256968 (0xf09f bytes) 000379[0] 1788199392.298032 mapread 0x85e11d thru 0x863499 (0x537d bytes) 000383[0] 1788199392.446964 trunc from 0x881964 to 0x00bccd (0x875c96 bytes) 000384[0] 1788199392.486186 write 0x54604b thru 0x54fdf5 (0x9dab bytes) 000385[0] 1788199392.498495 trunc from 0x54fdf6 to 0x81145f (0x2c166a bytes) 000390[0] 1788199392.744570 trunc from 0x811460 to 0x860712 (0x4f2b3 bytes) 000393[1] 1788199392.849887 read 0x3a3315 thru 0x3a5c27 (0x2913 bytes) 000394[1] 1788199392.886805 mapread 0x688513 thru 0x68c035 (0x3b23 bytes) 000396[1] 1788199393.008145 trunc from 0x860713 to 0x72a032 (0x1366e0 bytes) 000397[1] 1788199393.072286 read 0x5027e3 thru 0x504a44 (0x2262 bytes) truncating to largest ever: 0x9ff1ed 000398[1] 1788199393.078204 trunc from 0x72a033 to 0x9ff1ec (0x2d51ba bytes) 000399[1] 1788199393.114547 mapread 0x57cc0e thru 0x587a00 (0xadf3 bytes) 000401[1] 1788199393.220169 punch from 0x2aef0e to 0x2b0a31 (0x1b24 bytes) 000402[0] 1788199393.226128 mapread 0x3f5d1e thru 0x402dac (0xd08f bytes) 000404[1] 1788199393.386196 trunc from 0x9ff1ed to 0x98afeb (0x74201 bytes) 000407[1] 1788199393.569497 read 0x3c616d thru 0x3c893d (0x27d1 bytes) 000409[0] 1788199393.577304 trunc from 0x98afec to 0x55f778 (0x42b873 bytes) 000410[1] 1788199393.662445 mapread 0x04616e thru 0x04d945 (0x77d8 bytes) 000411[0] 1788199393.863562 mapwrite 0x964a80 thru 0x96fed8 (0xb459 bytes) 000412[1] 1788199394.262440 mapwrite 0x068c97 thru 0x06cfaf (0x4319 bytes) 000415[0] 1788199394.664723 mapread 0x80ea80 thru 0x81d2ff (0xe880 bytes) 000418[0] 1788199394.826330 trunc from 0x96fed9 to 0x11be80 (0x854058 bytes) 000420[0] 1788199394.925848 write 0x8c7935 thru 0x8c8540 (0xc0c bytes) 000422[1] 1788199394.940329 trunc from 0x8c8541 to 0x41408a (0x4b44b6 bytes) 000423[1] 1788199395.002670 trunc from 0x41408b to 0x616848 (0x2027be bytes) 000424[1] 1788199395.032337 read 0x115045 thru 0x11e2fb (0x92b7 bytes) 000425[0] 1788199395.044764 write 0x3bb6bc thru 0x3bf2cc (0x3c11 bytes) 000427[0] 1788199395.157876 punch from 0x2d03f2 to 0x2db9fa (0xb609 bytes) 000429[1] 1788199395.249942 write 0x44280a thru 0x450f48 (0xe73f bytes) 000430[0] 1788199395.314341 write 0x25a851 thru 0x269864 (0xf014 bytes) 000431[1] 1788199395.389075 read 0x0fcd16 thru 0x107fc8 (0xb2b3 bytes) 000432[0] 1788199395.433910 read 0x3789e0 thru 0x37a214 (0x1835 bytes) 000433[0] 1788199395.452859 read 0x4038ad thru 0x412d6f (0xf4c3 bytes) 000436[0] 1788199395.541883 punch from 0x2ef6ac to 0x2ff2e8 (0xfc3d bytes) 000441[1] 1788199395.847485 mapwrite 0x6c57f7 thru 0x6d130b (0xbb15 bytes) 000442[0] 1788199396.117562 trunc from 0x6d130c to 0x2f4a4d (0x3dc8be bytes) 000443[0] 1788199396.283512 mapread 0x05912d thru 0x05bbc6 (0x2a9a bytes) 000444[1] 1788199396.431714 read 0x0599f2 thru 0x06947d (0xfa8c bytes) 000445[1] 1788199396.616696 punch from 0x0143fe to 0x01b94c (0x754f bytes) 000447[1] 1788199396.698108 trunc from 0x2f4a4e to 0x581a3f (0x28cff2 bytes) 000449[1] 1788199396.791205 write 0x4b513b thru 0x4b86b8 (0x357e bytes) 000450[1] 1788199396.907265 read 0x11dd9a thru 0x1202b4 (0x251b bytes) 000451[1] 1788199396.914332 read 0x15b4f2 thru 0x16880c (0xd31b bytes) 000453[0] 1788199396.947153 mapread 0x28e378 thru 0x28f4f6 (0x117f bytes) 000454[1] 1788199397.093094 mapwrite 0x4ffab0 thru 0x505907 (0x5e58 bytes) 000456[1] 1788199397.202566 write 0x8b4285 thru 0x8b47e0 (0x55c bytes) 000458[1] 1788199397.270976 mapwrite 0x28b931 thru 0x295334 (0x9a04 bytes) 000459[1] 1788199397.406731 write 0x1e7adf thru 0x1eba44 (0x3f66 bytes) 000460[1] 1788199397.441695 mapwrite 0x22caf1 thru 0x234c2d (0x813d bytes) 000462[1] 1788199397.591880 mapwrite 0x118841 thru 0x123902 (0xb0c2 bytes) 000463[1] 1788199397.788010 read 0x68f616 thru 0x68fdb4 (0x79f bytes) 000465[1] 1788199397.837003 punch from 0x753ccf to 0x75d5a6 (0x98d8 bytes) 000466[0] 1788199397.848772 punch from 0x3a5014 to 0x3ad061 (0x804e bytes) 000467[0] 1788199397.886667 punch from 0x6db160 to 0x6e9b49 (0xe9ea bytes) 000469[0] 1788199397.973167 read 0x5a1d9c thru 0x5ad674 (0xb8d9 bytes) 000470[1] 1788199398.127131 punch from 0x14be7d to 0x151dfe (0x5f82 bytes) 000471[1] 1788199398.171310 trunc from 0x8b47e1 to 0x6d68c5 (0x1ddf1b bytes) 000472[1] 1788199398.295282 mapwrite 0x334c93 thru 0x341898 (0xcc06 bytes) 000473[0] 1788199398.558961 punch from 0x411ab7 to 0x415d02 (0x424c bytes) 000475[0] 1788199398.670039 punch from 0x6a189a to 0x6a4b3b (0x32a2 bytes) 000476[1] 1788199398.710991 read 0x101135 thru 0x110138 (0xf004 bytes) 000477[0] 1788199398.752257 punch from 0x343700 to 0x34bdfd (0x86fe bytes) 000478[0] 1788199398.806899 read 0x327acf thru 0x332700 (0xac32 bytes) 000479[0] 1788199398.821303 read 0x627810 thru 0x6336a1 (0xbe92 bytes) 000480[0] 1788199398.916577 mapwrite 0x8492cb thru 0x85471b (0xb451 bytes) 000482[1] 1788199399.186603 write 0x86ca72 thru 0x877ed0 (0xb45f bytes) 000485[1] 1788199399.292908 punch from 0x25d4d1 to 0x2630a8 (0x5bd8 bytes) 000487[0] 1788199399.356825 write 0x834019 thru 0x83fa31 (0xba19 bytes) 000488[1] 1788199399.411120 mapread 0x1e08aa thru 0x1e7d27 (0x747e bytes) 000490[0] 1788199399.551077 read 0x3a2bf5 thru 0x3a982c (0x6c38 bytes) 000492[0] 1788199399.567240 write 0x707d87 thru 0x717d09 (0xff83 bytes) 000493[1] 1788199399.597267 trunc from 0x877ed1 to 0x4e45e3 (0x3938ed bytes) 000495[1] 1788199399.691246 read 0x2e7ef5 thru 0x2ee790 (0x689c bytes) 000496[1] 1788199399.707452 mapwrite 0x0db2b2 thru 0x0dc746 (0x1495 bytes) 000500[0] 1788199399.869273 write 0x7f8117 thru 0x801b36 (0x9a20 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 9400 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x5fc5bf 000001[1] 1788199401.223777 trunc from 00000000 to 0x5fc5be (0x5fc5bf bytes) 000003[1] 1788199401.254827 mapwrite 0x73e000 thru 0x74a1c2 (0xc1c3 bytes) 000006[0] 1788199401.524903 trunc from 0x74a1c3 to 0x0fa61c (0x64fba6 bytes) 000008[0] 1788199401.595575 punch from 0x073ce4 to 0x07c37e (0x869b bytes) 000009[0] 1788199401.604681 write 0x354000 thru 0x3602b2 (0xc2b3 bytes) 000010[1] 1788199401.623153 read 0x0ea000 thru 0x0f7ef7 (0xdef8 bytes) 000011[0] 1788199401.705974 trunc from 0x3602b3 to 0x03ec63 (0x32164f bytes) 000012[1] 1788199401.801591 mapwrite 0x277000 thru 0x286e40 (0xfe41 bytes) 000014[1] 1788199402.069070 trunc from 0x286e41 to 0x4cc43f (0x2455ff bytes) truncating to largest ever: 0x67b3a4 000015[0] 1788199402.107143 trunc from 0x4cc440 to 0x67b3a3 (0x1aef64 bytes) 000016[0] 1788199402.171502 mapread 0x1bf000 thru 0x1c24ee (0x34ef bytes) 000018[0] 1788199402.231979 punch from 0x01b1dd to 0x01f725 (0x4549 bytes) 000019[1] 1788199402.238961 write 0x27a000 thru 0x286e47 (0xce48 bytes) 000021[1] 1788199402.318940 write 0x51e000 thru 0x52c2ee (0xe2ef bytes) 000022[0] 1788199402.343918 trunc from 0x67b3a4 to 0x261be9 (0x4197ba bytes) 000023[1] 1788199402.468781 mapwrite 0x5af000 thru 0x5b833c (0x933d bytes) 000028[1] 1788199402.627546 mapread 0x30d000 thru 0x317a01 (0xaa02 bytes) 000032[0] 1788199402.698833 read 0x432000 thru 0x43465b (0x265c bytes) 000033[1] 1788199402.728980 mapwrite 0x261000 thru 0x26d45f (0xc460 bytes) 000034[1] 1788199402.871722 write 0x085000 thru 0x092294 (0xd295 bytes) 000035[1] 1788199402.914291 trunc from 0x5b833d to 0x4e1c74 (0xd66c8 bytes) 000037[1] 1788199402.960708 read 0x291000 thru 0x2947a5 (0x37a6 bytes) 000038[1] 1788199402.968446 read 0x435000 thru 0x43a8f5 (0x58f6 bytes) 000039[1] 1788199403.037005 read 0x118000 thru 0x11fe08 (0x7e09 bytes) 000040[1] 1788199403.119350 read 0x08b000 thru 0x093900 (0x8901 bytes) 000041[0] 1788199403.178279 trunc from 0x4e1c75 to 0x217bbb (0x2ca0b9 bytes) 000042[0] 1788199403.354184 read 0x0bc000 thru 0x0c7c92 (0xbc93 bytes) 000043[1] 1788199403.476417 trunc from 0x217bbc to 0x207512 (0x106a9 bytes) 000044[0] 1788199403.524558 read 0x19c000 thru 0x1a475c (0x875d bytes) 000045[1] 1788199403.645654 trunc from 0x207513 to 0x0b9bfe (0x14d914 bytes) truncating to largest ever: 0x714a9f 000046[1] 1788199403.731819 trunc from 0x0b9bff to 0x714a9e (0x65aea0 bytes) 000050[1] 1788199403.845589 mapwrite 0x673000 thru 0x67b07e (0x807f bytes) 000051[0] 1788199404.079578 mapwrite 0x42e000 thru 0x435951 (0x7952 bytes) 000052[0] 1788199404.342530 write 0x123000 thru 0x12adb4 (0x7db5 bytes) 000054[1] 1788199404.356923 mapread 0x5a9000 thru 0x5b264c (0x964d bytes) 000055[1] 1788199404.459814 mapread 0x229000 thru 0x22caa5 (0x3aa6 bytes) 000062[0] 1788199404.493162 punch from 0x63d4f3 to 0x63e1f1 (0xcff bytes) 000063[0] 1788199404.522327 mapread 0x4a8000 thru 0x4af7da (0x77db bytes) 000065[1] 1788199404.577412 write 0x260000 thru 0x26d93a (0xd93b bytes) 000066[0] 1788199404.625443 punch from 0x31a47a to 0x3262a1 (0xbe28 bytes) 000067[0] 1788199404.701833 write 0x11c000 thru 0x127133 (0xb134 bytes) 000068[1] 1788199404.710532 read 0x381000 thru 0x384a9f (0x3aa0 bytes) 000070[0] 1788199404.764841 mapread 0x05a000 thru 0x05aa1a (0xa1b bytes) 000071[1] 1788199404.772576 trunc from 0x714a9f to 0x6cc544 (0x4855a bytes) 000074[0] 1788199404.869219 mapwrite 0x9ee000 thru 0x9f4e1b (0x6e1c bytes) 000075[1] 1788199405.111594 punch from 0x414865 to 0x4181b0 (0x394c bytes) 000076[1] 1788199405.137423 trunc from 0x9f4e1c to 0x6556d1 (0x39f74a bytes) 000078[1] 1788199405.199840 read 0x06d000 thru 0x07c37a (0xf37b bytes) 000079[1] 1788199405.209888 trunc from 0x6556d2 to 0x1156a1 (0x540030 bytes) 000081[1] 1788199405.295214 read 0x0c7000 thru 0x0d4af5 (0xdaf6 bytes) 000083[0] 1788199405.393979 mapwrite 0x58c000 thru 0x5963a5 (0xa3a6 bytes) 000085[1] 1788199405.584080 mapwrite 0x2a0000 thru 0x2a5aae (0x5aaf bytes) 000087[0] 1788199405.739855 mapwrite 0x522000 thru 0x52e3a8 (0xc3a9 bytes) 000088[1] 1788199405.951160 punch from 0x144285 to 0x14507e (0xdfa bytes) 000089[0] 1788199406.030062 mapread 0x166000 thru 0x16d032 (0x7033 bytes) 000090[1] 1788199406.112693 mapread 0x3e8000 thru 0x3ea717 (0x2718 bytes) 000091[0] 1788199406.146097 write 0x86f000 thru 0x8780b6 (0x90b7 bytes) 000092[1] 1788199406.203187 mapwrite 0x533000 thru 0x538938 (0x5939 bytes) 000094[1] 1788199406.435684 mapread 0x572000 thru 0x576cb8 (0x4cb9 bytes) 000097[1] 1788199406.464446 write 0x874000 thru 0x87859f (0x45a0 bytes) 000098[1] 1788199406.465403 read 0x52b000 thru 0x5347bb (0x97bc bytes) 000099[1] 1788199406.484109 punch from 0x4e12f5 to 0x4e97a6 (0x84b2 bytes) 000101[0] 1788199406.529825 write 0x013000 thru 0x01790d (0x490e bytes) 000103[1] 1788199406.562082 trunc from 0x8785a0 to 0x0bbe17 (0x7bc788 bytes) 000104[0] 1788199406.627307 write 0x4c6000 thru 0x4ca970 (0x4971 bytes) 000105[1] 1788199406.639069 punch from 0x173626 to 0x1823af (0xed8a bytes) 000106[1] 1788199406.675123 write 0x2d2000 thru 0x2deba6 (0xcba7 bytes) 000107[0] 1788199406.693128 mapwrite 0x9b5000 thru 0x9bc4d9 (0x74da bytes) 000108[0] 1788199406.893735 trunc from 0x9bc4da to 0x46b1c1 (0x551318 bytes) 000109[0] 1788199406.923038 mapwrite 0x009000 thru 0x00f8ef (0x68f0 bytes) 000111[1] 1788199407.050719 punch from 0x4441e9 to 0x44f97a (0xb792 bytes) 000113[1] 1788199407.085345 punch from 0x0e8a9b to 0x0f7219 (0xe77f bytes) 000116[1] 1788199407.113661 punch from 0x15474d to 0x16284c (0xe100 bytes) 000119[1] 1788199407.131974 mapwrite 0x4dd000 thru 0x4ea560 (0xd561 bytes) 000120[1] 1788199407.257159 punch from 0x233a51 to 0x238612 (0x4bc2 bytes) 000122[1] 1788199407.285294 punch from 0x2ac411 to 0x2b911c (0xcd0c bytes) 000123[1] 1788199407.290045 read 0x188000 thru 0x18b5e2 (0x35e3 bytes) 000124[1] 1788199407.298381 mapwrite 0x4f6000 thru 0x502765 (0xc766 bytes) 000125[1] 1788199407.426118 write 0x7a2000 thru 0x7a8cca (0x6ccb bytes) 000126[0] 1788199407.452149 mapwrite 0x6c8000 thru 0x6cfced (0x7cee bytes) 000131[1] 1788199407.649860 mapwrite 0x89c000 thru 0x8a477b (0x877c bytes) 000132[0] 1788199407.793103 punch from 0x48571f to 0x4943e9 (0xeccb bytes) 000136[0] 1788199407.848896 trunc from 0x8a477c to 0x3526ed (0x55208e bytes) 000137[1] 1788199407.910471 punch from 0x23df70 to 0x244e16 (0x6ea7 bytes) 000138[0] 1788199407.943459 mapread 0x281000 thru 0x290e9d (0xfe9e bytes) 000139[1] 1788199408.136379 trunc from 0x3526ee to 0x1f58f8 (0x15cdf5 bytes) 000140[0] 1788199408.177481 read 0x168000 thru 0x170e99 (0x8e9a bytes) 000143[0] 1788199408.292398 write 0x004000 thru 0x007c98 (0x3c99 bytes) 000144[0] 1788199408.329732 read 0x0ae000 thru 0x0b342f (0x5430 bytes) 000145[0] 1788199408.435002 trunc from 0x1f58f9 to 0x692e05 (0x49d50d bytes) 000146[0] 1788199408.475370 mapread 0x1e8000 thru 0x1f7559 (0xf55a bytes) 000148[1] 1788199408.538129 mapread 0x49e000 thru 0x4a6893 (0x8894 bytes) 000149[1] 1788199408.581008 mapread 0x1a5000 thru 0x1a96f4 (0x46f5 bytes) truncating to largest ever: 0x873267 000150[0] 1788199408.604235 trunc from 0x692e06 to 0x873266 (0x1e0461 bytes) 000151[1] 1788199408.634428 mapwrite 0x13c000 thru 0x140cb6 (0x4cb7 bytes) 000153[0] 1788199408.780475 punch from 0x0e5195 to 0x0ed32b (0x8197 bytes) 000154[1] 1788199408.838139 trunc from 0x873267 to 0x616a08 (0x25c85e bytes) 000156[0] 1788199408.898264 trunc from 0x616a09 to 0x82a0ae (0x2136a6 bytes) 000157[1] 1788199408.936811 mapread 0x184000 thru 0x187520 (0x3521 bytes) 000158[0] 1788199408.980010 mapwrite 0x8f2000 thru 0x8fca03 (0xaa04 bytes) 000160[0] 1788199409.207809 write 0x0aa000 thru 0x0ab185 (0x1186 bytes) 000162[0] 1788199409.231014 read 0x48c000 thru 0x491fd9 (0x5fda bytes) 000166[0] 1788199409.262204 read 0x699000 thru 0x6a45e3 (0xb5e4 bytes) 000167[0] 1788199409.262877 trunc from 0x8fca04 to 0x84a52a (0xb24d9 bytes) 000168[0] 1788199409.299325 read 0x06e000 thru 0x06eb69 (0xb6a bytes) 000170[1] 1788199409.321409 write 0x1f9000 thru 0x1fdae4 (0x4ae5 bytes) 000172[0] 1788199409.351419 write 0x506000 thru 0x511b42 (0xbb43 bytes) 000174[0] 1788199409.385482 mapread 0x690000 thru 0x69ad44 (0xad45 bytes) 000176[1] 1788199409.408337 mapwrite 0x509000 thru 0x517674 (0xe675 bytes) 000177[1] 1788199409.523436 mapread 0x048000 thru 0x056d6d (0xed6e bytes) 000178[0] 1788199409.553167 read 0x058000 thru 0x066183 (0xe184 bytes) 000180[0] 1788199409.577831 trunc from 0x84a52b to 0x7a545c (0xa50ce bytes) 000181[1] 1788199409.613147 mapread 0x72d000 thru 0x732d37 (0x5d38 bytes) 000182[0] 1788199409.650996 punch from 0x6b0be5 to 0x6b65b7 (0x59d3 bytes) 000183[1] 1788199409.670105 trunc from 0x7a545d to 0x2dc01d (0x4c943f bytes) 000184[0] 1788199409.717448 trunc from 0x2dc01e to 0x23803c (0xa3fe1 bytes) 000189[0] 1788199409.777103 trunc from 0x23803d to 0x2b1d45 (0x79d09 bytes) 000191[0] 1788199409.819840 mapread 0x136000 thru 0x140bdb (0xabdc bytes) truncating to largest ever: 0x97c48c 000192[1] 1788199409.852212 trunc from 0x2b1d46 to 0x97c48b (0x6ca746 bytes) 000193[0] 1788199409.890930 mapread 0x15c000 thru 0x15c633 (0x634 bytes) 000195[0] 1788199409.923475 mapread 0x4fa000 thru 0x506460 (0xc461 bytes) 000197[1] 1788199409.952877 mapread 0x8f6000 thru 0x8f877e (0x277f bytes) 000198[0] 1788199409.972061 mapread 0x955000 thru 0x964635 (0xf636 bytes) 000201[1] 1788199410.009571 mapread 0x798000 thru 0x7a68a1 (0xe8a2 bytes) 000204[1] 1788199410.057794 write 0x59c000 thru 0x59f9fa (0x39fb bytes) 000205[1] 1788199410.095486 mapwrite 0x1ee000 thru 0x1f8aeb (0xaaec bytes) 000206[0] 1788199410.230194 write 0x1a5000 thru 0x1acfed (0x7fee bytes) 000207[0] 1788199410.265475 punch from 0x1458a0 to 0x147619 (0x1d7a bytes) 000209[1] 1788199410.329779 write 0x768000 thru 0x76acdc (0x2cdd bytes) 000210[1] 1788199410.348954 trunc from 0x97c48c to 0x96c277 (0x10214 bytes) 000214[0] 1788199410.405877 trunc from 0x96c278 to 0x7fff55 (0x16c322 bytes) 000215[1] 1788199410.455101 write 0x737000 thru 0x7469ac (0xf9ad bytes) 000216[0] 1788199410.471552 trunc from 0x7fff56 to 0x16d89a (0x6926bb bytes) 000217[0] 1788199410.520685 mapwrite 0x42e000 thru 0x438a63 (0xaa64 bytes) 000219[0] 1788199410.725774 write 0x763000 thru 0x771a4a (0xea4b bytes) 000221[0] 1788199410.812173 punch from 0x31aedc to 0x323651 (0x8776 bytes) 000222[0] 1788199410.823923 write 0x802000 thru 0x80709a (0x509b bytes) 000224[1] 1788199410.906714 mapread 0x151000 thru 0x1535bb (0x25bc bytes) truncating to largest ever: 0x9ccd50 000225[0] 1788199411.032073 trunc from 0x80709b to 0x9ccd4f (0x1c5cb5 bytes) 000228[1] 1788199411.133680 mapwrite 0x05e000 thru 0x060c79 (0x2c7a bytes) 000229[1] 1788199411.295889 punch from 0x42a51c to 0x430f60 (0x6a45 bytes) 000230[0] 1788199411.352904 trunc from 0x9ccd50 to 0x37bb2a (0x651225 bytes) 000231[0] 1788199411.413034 mapread 0x1d9000 thru 0x1db4de (0x24df bytes) 000235[0] 1788199411.444388 mapwrite 0x118000 thru 0x126839 (0xe83a bytes) 000238[1] 1788199411.597795 trunc from 0x37bb2b to 0x50cc5d (0x191133 bytes) 000239[0] 1788199411.693275 punch from 0x233678 to 0x233f1e (0x8a7 bytes) 000240[1] 1788199411.762443 punch from 0x16a1c2 to 0x1752d8 (0xb117 bytes) 000241[1] 1788199411.820328 write 0x320000 thru 0x32cb56 (0xcb57 bytes) 000243[1] 1788199411.879700 mapread 0x435000 thru 0x43f920 (0xa921 bytes) 000245[0] 1788199411.972679 trunc from 0x50cc5e to 0x8e45c8 (0x3d796b bytes) 000246[0] 1788199412.014960 trunc from 0x8e45c9 to 0x3f3873 (0x4f0d55 bytes) 000248[0] 1788199412.034371 mapwrite 0x711000 thru 0x71d536 (0xc537 bytes) 000249[0] 1788199412.138111 punch from 0x3d0cc7 to 0x3d4aa0 (0x3dda bytes) 000252[0] 1788199412.146976 mapread 0x63b000 thru 0x640e2f (0x5e30 bytes) 000255[1] 1788199412.178975 mapread 0x3a9000 thru 0x3ac14b (0x314c bytes) 000258[0] 1788199412.211941 trunc from 0x71d537 to 0x867a4a (0x14a514 bytes) 000259[0] 1788199412.260087 trunc from 0x867a4b to 0x00b69d (0x85c3ad bytes) 000260[1] 1788199412.279232 mapwrite 0x528000 thru 0x536846 (0xe847 bytes) 000262[1] 1788199412.416698 read 0x525000 thru 0x532e7f (0xde80 bytes) 000263[1] 1788199412.417221 punch from 0x50b3da to 0x50bbaf (0x7d6 bytes) 000264[1] 1788199412.437305 read 0x15d000 thru 0x162c63 (0x5c64 bytes) 000266[1] 1788199412.483155 read 0x05d000 thru 0x068bbd (0xbbbe bytes) 000268[1] 1788199412.511271 write 0x668000 thru 0x67298f (0xa990 bytes) 000270[0] 1788199412.542533 trunc from 0x672990 to 0x99c0f2 (0x329763 bytes) 000274[1] 1788199412.615831 read 0x015000 thru 0x01bed9 (0x6eda bytes) 000275[1] 1788199412.638972 write 0x6f9000 thru 0x7033de (0xa3df bytes) 000276[0] 1788199412.654277 read 0x393000 thru 0x39fb2a (0xcb2b bytes) 000278[1] 1788199412.678428 mapread 0x860000 thru 0x86a784 (0xa785 bytes) 000281[1] 1788199412.695980 write 0x7f9000 thru 0x8063f0 (0xd3f1 bytes) 000282[0] 1788199412.716923 mapread 0x3d1000 thru 0x3da3ca (0x93cb bytes) 000283[0] 1788199412.777724 punch from 0x4c89d6 to 0x4cf8ff (0x6f2a bytes) 000285[1] 1788199412.817879 write 0x202000 thru 0x20f37e (0xd37f bytes) 000286[0] 1788199412.855151 punch from 0x035afb to 0x03d347 (0x784d bytes) 000287[1] 1788199412.881576 mapwrite 0x615000 thru 0x61fb7b (0xab7c bytes) 000289[0] 1788199412.987758 read 0x220000 thru 0x222514 (0x2515 bytes) 000290[1] 1788199413.023324 mapread 0x5dd000 thru 0x5e0dca (0x3dcb bytes) 000292[1] 1788199413.051022 read 0x823000 thru 0x82d140 (0xa141 bytes) 000297[0] 1788199413.066473 mapwrite 0x473000 thru 0x47d764 (0xa765 bytes) 000298[1] 1788199413.160915 read 0x395000 thru 0x3a049a (0xb49b bytes) 000299[0] 1788199413.188573 read 0x5ea000 thru 0x5f46a7 (0xa6a8 bytes) truncating to largest ever: 0x9e4a13 000301[1] 1788199413.197324 trunc from 0x99c0f3 to 0x9e4a12 (0x48920 bytes) 000302[1] 1788199413.231347 mapread 0x627000 thru 0x62d1a9 (0x61aa bytes) 000303[1] 1788199413.264909 mapwrite 0x35e000 thru 0x36cdee (0xedef bytes) 000305[1] 1788199413.336311 mapwrite 0x606000 thru 0x6138af (0xd8b0 bytes) 000306[1] 1788199413.406806 mapread 0x90e000 thru 0x9162a9 (0x82aa bytes) 000307[0] 1788199413.439227 mapwrite 0x0c7000 thru 0x0d57c7 (0xe7c8 bytes) 000308[1] 1788199413.588001 trunc from 0x9e4a13 to 0x950a1a (0x93ff8 bytes) 000309[1] 1788199413.621332 write 0x44d000 thru 0x44ec5f (0x1c60 bytes) 000310[0] 1788199413.627443 write 0x185000 thru 0x1890f1 (0x40f2 bytes) 000313[1] 1788199413.653931 read 0x053000 thru 0x059d35 (0x6d36 bytes) 000314[1] 1788199413.683982 punch from 0x079dd2 to 0x07cebf (0x30ee bytes) 000315[1] 1788199413.711244 trunc from 0x950a1b to 0x9c7b3c (0x77122 bytes) 000316[1] 1788199413.738856 punch from 0x6406b1 to 0x648f90 (0x88e0 bytes) 000321[1] 1788199413.776019 trunc from 0x9c7b3d to 0x181000 (0x846b3c bytes) 000322[0] 1788199413.796391 read 0x093000 thru 0x096e91 (0x3e92 bytes) 000327[1] 1788199413.829360 write 0x617000 thru 0x6174cd (0x4ce bytes) 000330[0] 1788199413.855174 write 0x953000 thru 0x9558d2 (0x28d3 bytes) 000331[1] 1788199413.875788 trunc from 0x9558d3 to 0x722fe4 (0x2328ee bytes) 000334[0] 1788199413.954859 punch from 0x226680 to 0x22e266 (0x7be7 bytes) 000336[1] 1788199414.000612 read 0x19b000 thru 0x19f281 (0x4282 bytes) 000338[1] 1788199414.026146 punch from 0x0c26da to 0x0cbc8e (0x95b5 bytes) 000339[1] 1788199414.043851 read 0x105000 thru 0x10fd09 (0xad0a bytes) 000340[1] 1788199414.068314 trunc from 0x722fe5 to 0x071f89 (0x6b105b bytes) 000341[0] 1788199414.106785 mapwrite 0x779000 thru 0x782a3d (0x9a3e bytes) 000342[1] 1788199414.204214 write 0x9c0000 thru 0x9ccaf4 (0xcaf5 bytes) 000343[1] 1788199414.224284 punch from 0x8863da to 0x886f89 (0xbb0 bytes) 000344[1] 1788199414.244596 mapread 0x896000 thru 0x897a94 (0x1a95 bytes) 000345[1] 1788199414.263707 read 0x2fe000 thru 0x30dd3e (0xfd3f bytes) 000346[1] 1788199414.273564 mapwrite 0x3ac000 thru 0x3b849b (0xc49c bytes) 000347[0] 1788199414.358886 write 0x925000 thru 0x925e28 (0xe29 bytes) 000349[0] 1788199414.412416 trunc from 0x9ccaf5 to 0x35e822 (0x66e2d2 bytes) 000350[0] 1788199414.433599 mapread 0x2e1000 thru 0x2e7945 (0x6946 bytes) 000351[1] 1788199414.457323 write 0x5da000 thru 0x5e6dc9 (0xcdca bytes) 000352[0] 1788199414.489901 read 0x2af000 thru 0x2bab83 (0xbb84 bytes) 000354[0] 1788199414.619582 punch from 0x0beeb5 to 0x0c30bf (0x420b bytes) 000355[0] 1788199414.650813 mapread 0x13b000 thru 0x13e7ce (0x37cf bytes) 000358[1] 1788199414.695551 mapwrite 0x8bc000 thru 0x8c8947 (0xc948 bytes) 000359[0] 1788199414.854264 mapread 0x3fd000 thru 0x40c582 (0xf583 bytes) 000361[1] 1788199414.991435 write 0x558000 thru 0x567ffb (0xfffc bytes) 000364[1] 1788199415.034157 mapread 0x6cb000 thru 0x6d5cee (0xacef bytes) 000365[1] 1788199415.070650 punch from 0x6720d5 to 0x673dab (0x1cd7 bytes) 000366[0] 1788199415.074486 read 0x043000 thru 0x051910 (0xe911 bytes) 000370[0] 1788199415.112804 mapread 0x4cc000 thru 0x4d6269 (0xa26a bytes) 000371[1] 1788199415.131248 read 0x82a000 thru 0x82a0e7 (0xe8 bytes) 000373[1] 1788199415.136351 mapwrite 0x27d000 thru 0x2841a3 (0x71a4 bytes) 000374[1] 1788199415.250613 write 0x36b000 thru 0x36b902 (0x903 bytes) 000375[0] 1788199415.268807 punch from 0x35afb1 to 0x368674 (0xd6c4 bytes) 000376[0] 1788199415.289409 read 0x5dd000 thru 0x5e45c6 (0x75c7 bytes) 000377[1] 1788199415.299151 mapwrite 0x24f000 thru 0x25b49e (0xc49f bytes) 000378[1] 1788199415.394704 mapwrite 0x027000 thru 0x029174 (0x2175 bytes) 000381[1] 1788199415.455656 read 0x60a000 thru 0x613c6a (0x9c6b bytes) 000383[1] 1788199415.476594 trunc from 0x8c8948 to 0x177f7a (0x7509cd bytes) 000384[0] 1788199415.497879 mapwrite 0x919000 thru 0x921aee (0x8aef bytes) 000385[1] 1788199415.607774 mapread 0x3ef000 thru 0x3f0061 (0x1062 bytes) 000387[1] 1788199415.651343 read 0x520000 thru 0x52c951 (0xc952 bytes) 000388[0] 1788199415.663053 mapread 0x40c000 thru 0x4101f4 (0x41f5 bytes) 000390[1] 1788199415.689156 read 0x31b000 thru 0x32ac7d (0xfc7e bytes) 000391[0] 1788199415.712974 read 0x0d4000 thru 0x0dfa4c (0xba4d bytes) 000393[1] 1788199415.735666 trunc from 0x921aef to 0x462610 (0x4bf4de bytes) 000394[1] 1788199415.825885 read 0x295000 thru 0x296058 (0x1059 bytes) 000396[1] 1788199415.921665 mapwrite 0x205000 thru 0x205ace (0xacf bytes) 000397[1] 1788199416.047381 punch from 0x19aed7 to 0x1a062e (0x5758 bytes) 000398[0] 1788199416.060321 mapwrite 0x869000 thru 0x86a30d (0x130e bytes) 000400[0] 1788199416.191556 trunc from 0x86a30e to 0x25a533 (0x60fdda bytes) 000404[1] 1788199416.335493 read 0x1ea000 thru 0x1f3758 (0x9759 bytes) 000406[1] 1788199416.399982 trunc from 0x25a534 to 0x6cc209 (0x471cd6 bytes) 000407[1] 1788199416.459886 mapwrite 0x67f000 thru 0x689a7b (0xaa7c bytes) 000408[0] 1788199416.640210 mapread 0x593000 thru 0x59d25c (0xa25d bytes) 000409[0] 1788199416.699279 mapwrite 0x9c2000 thru 0x9c3d4d (0x1d4e bytes) 000410[0] 1788199416.806654 mapwrite 0x847000 thru 0x8569bf (0xf9c0 bytes) 000412[1] 1788199416.918006 read 0x97f000 thru 0x9890f8 (0xa0f9 bytes) 000414[1] 1788199416.948753 trunc from 0x9c3d4e to 0x4fb0a4 (0x4c8ca9 bytes) 000415[1] 1788199417.007902 trunc from 0x4fb0a5 to 0x08d24f (0x46de55 bytes) 000416[0] 1788199417.088837 mapwrite 0x29d000 thru 0x2aafbe (0xdfbf bytes) 000418[1] 1788199417.366602 trunc from 0x2aafbf to 0x22416a (0x86e54 bytes) 000419[0] 1788199417.418463 trunc from 0x22416b to 0x2007e4 (0x23986 bytes) 000421[0] 1788199417.483946 punch from 0x1877cd to 0x1968b7 (0xf0eb bytes) 000422[0] 1788199417.540142 trunc from 0x2007e5 to 0x7c328b (0x5c2aa7 bytes) 000423[0] 1788199417.627502 write 0x47d000 thru 0x483641 (0x6642 bytes) 000424[0] 1788199417.655493 trunc from 0x7c328c to 0x69409c (0x12f1ef bytes) 000425[1] 1788199417.705695 mapread 0x1bd000 thru 0x1c9a3c (0xca3d bytes) 000426[0] 1788199417.826727 write 0x7e6000 thru 0x7eff33 (0x9f34 bytes) 000428[1] 1788199417.901905 trunc from 0x7eff34 to 0x43a5ce (0x3b5965 bytes) 000429[0] 1788199418.023556 mapread 0x15a000 thru 0x164399 (0xa39a bytes) 000430[0] 1788199418.095181 punch from 0x1889e6 to 0x18cc28 (0x4243 bytes) 000431[1] 1788199418.144897 write 0x1b4000 thru 0x1bcaaa (0x8aab bytes) 000433[1] 1788199418.191464 write 0x8c5000 thru 0x8c5b7c (0xb7d bytes) 000436[0] 1788199418.246471 mapwrite 0x82f000 thru 0x8391de (0xa1df bytes) 000437[1] 1788199418.397272 write 0x3f8000 thru 0x40494a (0xc94b bytes) 000440[0] 1788199418.432938 write 0x673000 thru 0x6801ee (0xd1ef bytes) 000441[0] 1788199418.474910 punch from 0x6e95ff to 0x6ea877 (0x1279 bytes) 000443[1] 1788199418.502836 mapread 0x0cc000 thru 0x0dbf06 (0xff07 bytes) 000446[1] 1788199418.566762 write 0x9a3000 thru 0x9b0227 (0xd228 bytes) 000447[0] 1788199418.585703 trunc from 0x9b0228 to 0x1affd5 (0x800252 bytes) 000450[0] 1788199418.731310 punch from 0x15ff6a to 0x169258 (0x92ef bytes) 000453[1] 1788199418.753948 mapwrite 0x886000 thru 0x88d188 (0x7189 bytes) 000454[0] 1788199418.901644 mapwrite 0x87b000 thru 0x886c95 (0xbc96 bytes) 000456[1] 1788199419.027779 read 0x37c000 thru 0x3831da (0x71db bytes) 000457[0] 1788199419.058493 mapread 0x019000 thru 0x01ca3f (0x3a40 bytes) 000458[1] 1788199419.111485 read 0x829000 thru 0x837ddc (0xeddd bytes) 000459[0] 1788199419.124176 mapwrite 0x5e8000 thru 0x5f0edf (0x8ee0 bytes) 000462[0] 1788199419.315925 mapread 0x5f8000 thru 0x5f92d4 (0x12d5 bytes) 000463[1] 1788199419.348600 mapread 0x322000 thru 0x328e4b (0x6e4c bytes) 000465[1] 1788199419.414233 mapwrite 0x6de000 thru 0x6e9206 (0xb207 bytes) 000468[1] 1788199419.585064 mapread 0x1a3000 thru 0x1a475d (0x175e bytes) 000469[0] 1788199419.644736 mapread 0x016000 thru 0x01aa4a (0x4a4b bytes) 000470[1] 1788199419.736052 mapwrite 0x8f2000 thru 0x90166b (0xf66c bytes) 000471[0] 1788199419.974876 read 0x397000 thru 0x3a5e87 (0xee88 bytes) 000473[1] 1788199420.036999 write 0x5e3000 thru 0x5f009f (0xd0a0 bytes) 000474[1] 1788199420.133912 punch from 0x7837ba to 0x78a4e9 (0x6d30 bytes) 000476[0] 1788199420.184433 trunc from 0x90166c to 0x8cb0db (0x36590 bytes) 000478[1] 1788199420.269935 mapwrite 0x7ee000 thru 0x7f1c50 (0x3c51 bytes) 000479[1] 1788199420.350438 punch from 0x75f815 to 0x768eeb (0x96d7 bytes) 000480[1] 1788199420.354612 punch from 0x64b0b1 to 0x6596b5 (0xe605 bytes) 000481[1] 1788199420.378017 punch from 0x0c42ff to 0x0c746f (0x3171 bytes) 000482[1] 1788199420.382394 read 0x632000 thru 0x63e362 (0xc363 bytes) 000483[1] 1788199420.398185 punch from 0x6e59b2 to 0x6ef2e4 (0x9933 bytes) 000484[0] 1788199420.415744 mapwrite 0x0a5000 thru 0x0adb7f (0x8b80 bytes) 000485[0] 1788199420.516917 read 0x6c8000 thru 0x6d2848 (0xa849 bytes) 000490[0] 1788199420.592310 write 0x047000 thru 0x051ed7 (0xaed8 bytes) 000491[1] 1788199420.613055 write 0x1ba000 thru 0x1bbfee (0x1fef bytes) 000493[0] 1788199420.675536 mapwrite 0x7d1000 thru 0x7d413a (0x313b bytes) 000495[0] 1788199420.820523 write 0x1c2000 thru 0x1c76b9 (0x56ba bytes) 000496[1] 1788199420.827944 mapread 0x484000 thru 0x487b07 (0x3b08 bytes) 000497[0] 1788199420.877827 trunc from 0x8cb0dc to 0x5912ca (0x339e11 bytes) 000499[1] 1788199420.918858 mapread 0x029000 thru 0x02efb9 (0x5fba bytes) All operations completed A-OK! PASS 16a (69s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 14:03:48 (1788199428) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.586458 s, 14.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.535622 s, 15.7 MB/s Chance of close/open is 1 in 50 Seed set to 9432 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003[1] 1788199433.381486 mapwrite 0x819021 thru 0x81a1d4 (0x11b4 bytes) 000005[0] 1788199433.648437 mapwrite 0x98e263 thru 0x990077 (0x1e15 bytes) 000007[1] 1788199433.911749 mapread 0x05a548 thru 0x05a94a (0x403 bytes) 000009[0] 1788199434.017148 mapwrite 0x6e5547 thru 0x6e5deb (0x8a5 bytes) 000010[0] 1788199434.091898 mapwrite 0x914114 thru 0x915f1d (0x1e0a bytes) 000012[1] 1788199434.205854 read 0x59d2cf thru 0x59d5c4 (0x2f6 bytes) 000015[1] 1788199434.228517 read 0x3bbeac thru 0x3bc549 (0x69e bytes) 000016[0] 1788199434.251804 punch from 0x6ebf64 to 0x6ecab5 (0xb52 bytes) 000017[1] 1788199434.270310 mapwrite 0x0aa282 thru 0x0ab0ea (0xe69 bytes) 000019[1] 1788199434.377503 write 0x23dfad thru 0x23ea71 (0xac5 bytes) truncating to largest ever: 0x67d822 000020[0] 1788199434.403516 trunc from 0x990078 to 0x67d821 (0x312856 bytes) 000021[1] 1788199434.465376 mapread 0x3cf0f8 thru 0x3d0908 (0x1811 bytes) 000023[0] 1788199434.607434 trunc from 0x67d822 to 0x0e7814 (0x59600d bytes) truncating to largest ever: 0x70b945 000032[0] 1788199434.785810 trunc from 0x0e7815 to 0x70b944 (0x624130 bytes) 000033[1] 1788199434.898740 write 0x933520 thru 0x933c7d (0x75e bytes) 000034[1] 1788199434.949942 punch from 0x3d6e54 to 0x3d7cc8 (0xe75 bytes) 000037[1] 1788199435.065027 mapwrite 0x981f95 thru 0x982e99 (0xf05 bytes) 000038[0] 1788199435.259574 mapwrite 0x125466 thru 0x127071 (0x1c0c bytes) 000039[1] 1788199435.367589 write 0x0e63aa thru 0x0e71c3 (0xe1a bytes) 000040[0] 1788199435.437230 read 0x744d2e thru 0x745950 (0xc23 bytes) 000041[0] 1788199435.494068 write 0x3281e6 thru 0x328c14 (0xa2f bytes) 000042[0] 1788199435.523484 punch from 0x4ba928 to 0x4bc2aa (0x1983 bytes) 000044[1] 1788199435.580807 punch from 0x91de3d to 0x91ecd3 (0xe97 bytes) 000045[1] 1788199435.628335 mapread 0x770533 thru 0x770a23 (0x4f1 bytes) 000048[0] 1788199435.689514 read 0x8bbef1 thru 0x8bda11 (0x1b21 bytes) 000049[1] 1788199435.716386 mapread 0x2fc677 thru 0x2fd4d9 (0xe63 bytes) 000051[0] 1788199435.736775 trunc from 0x982e9a to 0x2efcb5 (0x6931e4 bytes) 000053[1] 1788199435.868121 punch from 0x154870 to 0x154bea (0x37b bytes) 000054[0] 1788199435.956750 trunc from 0x2efcb6 to 0x349b04 (0x59e4f bytes) 000055[0] 1788199436.037232 mapwrite 0x5cd925 thru 0x5cdd75 (0x451 bytes) 000059[0] 1788199436.152458 punch from 0x3fb771 to 0x3fcca3 (0x1533 bytes) 000060[1] 1788199436.157528 read 0x096ea7 thru 0x097013 (0x16d bytes) 000061[0] 1788199436.196768 mapread 0x5a65f4 thru 0x5a6d60 (0x76d bytes) 000063[1] 1788199436.210482 mapwrite 0x9fdeb9 thru 0x9feb93 (0xcdb bytes) 000065[1] 1788199436.401308 punch from 0x1d052e to 0x1d10a4 (0xb77 bytes) 000066[1] 1788199436.410429 write 0x1ed170 thru 0x1eede6 (0x1c77 bytes) 000067[0] 1788199436.441842 write 0x4d320c thru 0x4d49c6 (0x17bb bytes) 000072[1] 1788199436.547408 punch from 0x1b4080 to 0x1b4458 (0x3d9 bytes) 000073[1] 1788199436.570824 mapread 0x10efe7 thru 0x10f4e1 (0x4fb bytes) 000075[0] 1788199436.616815 trunc from 0x9feb94 to 0x25f58a (0x79f609 bytes) 000077[1] 1788199436.681232 trunc from 0x25f58b to 0x228e19 (0x36771 bytes) 000078[0] 1788199436.732809 mapwrite 0x8cf6d5 thru 0x8d145b (0x1d87 bytes) 000079[1] 1788199436.886727 punch from 0x7f8291 to 0x7f9ce2 (0x1a52 bytes) 000080[0] 1788199436.925779 trunc from 0x8d145c to 0x36383a (0x56dc21 bytes) 000081[1] 1788199436.960507 read 0x042b32 thru 0x0432c3 (0x792 bytes) 000082[1] 1788199437.014293 read 0x171700 thru 0x171f1a (0x81b bytes) truncating to largest ever: 0x74fea9 000083[0] 1788199437.173511 trunc from 0x36383b to 0x74fea8 (0x3ec66e bytes) 000084[0] 1788199437.266393 trunc from 0x74fea9 to 0x3cf2da (0x380bce bytes) 000085[0] 1788199437.325120 read 0x3835cd thru 0x383dcf (0x803 bytes) 000086[1] 1788199437.403433 write 0x329ccb thru 0x32a0a6 (0x3dc bytes) 000087[1] 1788199437.541976 mapwrite 0x882816 thru 0x884265 (0x1a50 bytes) 000088[0] 1788199437.936780 trunc from 0x884266 to 0x430620 (0x453c45 bytes) 000091[1] 1788199438.081534 mapread 0x0bf1b2 thru 0x0bfd91 (0xbe0 bytes) 000092[0] 1788199438.123606 mapwrite 0x24e8d1 thru 0x250796 (0x1ec6 bytes) 000095[1] 1788199438.233327 write 0x605f06 thru 0x60601f (0x11a bytes) truncating to largest ever: 0x881572 000096[1] 1788199438.266785 trunc from 0x606020 to 0x881571 (0x27b552 bytes) 000099[1] 1788199438.308972 punch from 0x875ade to 0x876734 (0xc57 bytes) 000101[1] 1788199438.340504 mapread 0x034caa thru 0x034d7f (0xd6 bytes) 000104[1] 1788199438.372956 read 0x7fdcf1 thru 0x7ff5b5 (0x18c5 bytes) 000106[0] 1788199438.380228 read 0x4548fd thru 0x454e09 (0x50d bytes) 000109[0] 1788199438.457124 trunc from 0x881572 to 0x4e6b9e (0x39a9d3 bytes) 000111[0] 1788199438.540421 write 0x79723e thru 0x798275 (0x1038 bytes) 000112[0] 1788199438.542081 mapwrite 0x8bf240 thru 0x8c0ede (0x1c9f bytes) 000113[0] 1788199438.616961 trunc from 0x8c0edf to 0x74ad19 (0x1761c5 bytes) 000114[0] 1788199438.634605 write 0x822f98 thru 0x824306 (0x136f bytes) 000116[1] 1788199438.661359 mapread 0x013b5f thru 0x014c1c (0x10be bytes) 000117[1] 1788199438.698789 mapwrite 0x98baaf thru 0x98d9aa (0x1efc bytes) 000118[1] 1788199438.820888 mapread 0x276f02 thru 0x277bbe (0xcbd bytes) 000121[1] 1788199438.945564 mapwrite 0x01f922 thru 0x020f2a (0x1609 bytes) 000123[1] 1788199439.037307 write 0x4a1ed5 thru 0x4a26d1 (0x7fd bytes) 000126[1] 1788199439.110936 mapwrite 0x709c47 thru 0x70ad77 (0x1131 bytes) 000128[1] 1788199439.274438 mapread 0x1e61e0 thru 0x1e705a (0xe7b bytes) 000130[0] 1788199439.320470 read 0x6a6220 thru 0x6a7405 (0x11e6 bytes) 000133[1] 1788199439.378583 trunc from 0x98d9ab to 0x7b0737 (0x1dd273 bytes) 000134[1] 1788199439.433077 trunc from 0x7b0738 to 0x1f3267 (0x5bd4d0 bytes) 000135[0] 1788199439.470333 read 0x068aed thru 0x069c93 (0x11a7 bytes) 000136[1] 1788199439.522497 mapread 0x16328b thru 0x1632b0 (0x26 bytes) 000139[0] 1788199439.599218 punch from 0x0edea8 to 0x0edf93 (0xec bytes) 000140[0] 1788199439.647266 write 0x97aec9 thru 0x97c3f6 (0x152e bytes) truncating to largest ever: 0x930303 000142[1] 1788199439.717548 trunc from 0x97c3f7 to 0x930302 (0x4c0f4 bytes) 000143[0] 1788199439.800863 punch from 0x00d421 to 0x00e5ac (0x118c bytes) 000145[0] 1788199439.883532 mapwrite 0x2036ef thru 0x204791 (0x10a3 bytes) 000146[1] 1788199440.023300 mapwrite 0x17615d thru 0x176b6c (0xa10 bytes) 000148[1] 1788199440.261313 mapread 0x37f86a thru 0x380e8f (0x1626 bytes) 000150[0] 1788199440.352941 write 0x808e2c thru 0x80a6f4 (0x18c9 bytes) truncating to largest ever: 0x943210 000151[0] 1788199440.440795 trunc from 0x930303 to 0x94320f (0x12f0d bytes) 000152[0] 1788199440.495853 read 0x9107b0 thru 0x910ec7 (0x718 bytes) 000153[0] 1788199440.504520 mapwrite 0x94a116 thru 0x94b7c3 (0x16ae bytes) 000155[0] 1788199440.653156 read 0x21af7e thru 0x21b07b (0xfe bytes) 000156[0] 1788199440.699182 mapwrite 0x887d5f thru 0x888343 (0x5e5 bytes) 000158[1] 1788199440.771115 mapread 0x7b1c64 thru 0x7b3684 (0x1a21 bytes) 000160[1] 1788199440.989195 trunc from 0x94b7c4 to 0x34665f (0x605164 bytes) 000161[0] 1788199441.177339 read 0x310184 thru 0x311b9e (0x1a1b bytes) truncating to largest ever: 0x9623db 000162[0] 1788199441.414209 trunc from 0x346660 to 0x9623da (0x61bd7b bytes) 000164[1] 1788199441.508829 read 0x2f3aa0 thru 0x2f53a2 (0x1903 bytes) 000165[0] 1788199441.542795 write 0x22d395 thru 0x22f235 (0x1ea1 bytes) 000167[0] 1788199441.591010 punch from 0x314462 to 0x314d19 (0x8b8 bytes) 000170[0] 1788199441.615340 read 0x127e88 thru 0x129938 (0x1ab1 bytes) 000171[1] 1788199441.644200 punch from 0x311ec9 to 0x3126cd (0x805 bytes) 000173[0] 1788199441.694294 trunc from 0x9623db to 0x6685be (0x2f9e1c bytes) 000176[0] 1788199441.758135 read 0x2614b8 thru 0x261f3b (0xa84 bytes) 000178[0] 1788199441.791683 trunc from 0x6685bf to 0x2e8938 (0x37fc86 bytes) 000180[0] 1788199441.839500 mapwrite 0x640d12 thru 0x640f57 (0x246 bytes) 000181[0] 1788199441.948076 write 0x47dcf9 thru 0x47fa44 (0x1d4c bytes) 000182[1] 1788199442.001084 mapread 0x3becbc thru 0x3c044a (0x178f bytes) 000183[0] 1788199442.066217 mapwrite 0x17b907 thru 0x17cc85 (0x137f bytes) 000185[0] 1788199442.151808 punch from 0x1de05f to 0x1de122 (0xc4 bytes) 000186[1] 1788199442.157344 write 0x0a7feb thru 0x0a8ea4 (0xeba bytes) 000187[0] 1788199442.191079 mapread 0x4dad67 thru 0x4db0b5 (0x34f bytes) 000189[0] 1788199442.245861 mapread 0x570305 thru 0x571941 (0x163d bytes) 000190[0] 1788199442.282103 write 0x8b7b89 thru 0x8b879a (0xc12 bytes) 000191[1] 1788199442.331075 mapwrite 0x0b2e14 thru 0x0b3824 (0xa11 bytes) 000194[0] 1788199442.461791 mapread 0x4c87e1 thru 0x4c98c1 (0x10e1 bytes) 000195[0] 1788199442.494844 punch from 0x369d9d to 0x36bac2 (0x1d26 bytes) 000196[1] 1788199442.548961 mapread 0x595112 thru 0x5959f4 (0x8e3 bytes) 000198[1] 1788199442.626560 punch from 0x5dc605 to 0x5dd566 (0xf62 bytes) 000199[1] 1788199442.659369 mapwrite 0x156413 thru 0x1581ac (0x1d9a bytes) 000200[0] 1788199442.784117 write 0x2336da thru 0x233b3a (0x461 bytes) 000201[0] 1788199442.814801 punch from 0x87f542 to 0x880897 (0x1356 bytes) 000205[0] 1788199442.888673 mapwrite 0x767317 thru 0x768c41 (0x192b bytes) 000208[1] 1788199443.063690 mapwrite 0x9c8f62 thru 0x9c937e (0x41d bytes) 000213[1] 1788199443.253819 punch from 0x22ad66 to 0x22c69d (0x1938 bytes) 000214[0] 1788199443.259512 mapread 0x6c8f7b thru 0x6c90db (0x161 bytes) 000215[0] 1788199443.331385 mapread 0x7ad698 thru 0x7ad9d6 (0x33f bytes) 000218[0] 1788199443.383916 trunc from 0x9c937f to 0x20cbf8 (0x7bc786 bytes) 000219[0] 1788199443.438298 write 0x2b4b9b thru 0x2b621b (0x1681 bytes) 000220[1] 1788199443.442420 write 0x6161f8 thru 0x617ce9 (0x1af2 bytes) 000222[0] 1788199443.484485 trunc from 0x617cea to 0x5e62a0 (0x31a49 bytes) 000224[0] 1788199443.521447 mapwrite 0x3d8039 thru 0x3d9734 (0x16fc bytes) 000225[1] 1788199443.605212 read 0x39ebf1 thru 0x39ede2 (0x1f2 bytes) 000226[1] 1788199443.643009 trunc from 0x5e62a1 to 0x7e5f45 (0x1ffca5 bytes) 000227[0] 1788199443.701282 mapwrite 0x09eee5 thru 0x09ff8b (0x10a7 bytes) 000228[0] 1788199443.858963 mapwrite 0x31e5c5 thru 0x31eb9e (0x5da bytes) 000229[0] 1788199444.041804 mapwrite 0x8f3859 thru 0x8f3a6a (0x212 bytes) 000230[0] 1788199444.238958 write 0x2c4a02 thru 0x2c594b (0xf4a bytes) 000231[0] 1788199444.263097 punch from 0x4168bc to 0x418419 (0x1b5e bytes) 000234[0] 1788199444.332142 mapread 0x6c8fb5 thru 0x6c9214 (0x260 bytes) 000236[1] 1788199444.358853 mapread 0x4822d8 thru 0x483c4d (0x1976 bytes) 000237[1] 1788199444.405667 punch from 0x744fc2 to 0x745769 (0x7a8 bytes) 000238[0] 1788199444.435861 mapwrite 0x446d51 thru 0x448969 (0x1c19 bytes) 000239[0] 1788199444.559768 mapread 0x001665 thru 0x0030bc (0x1a58 bytes) 000240[0] 1788199444.604923 mapread 0x60b8de thru 0x60ba29 (0x14c bytes) 000242[0] 1788199444.613732 trunc from 0x8f3a6b to 0x055e6f (0x89dbfb bytes) 000246[0] 1788199444.696310 mapread 0x017fdc thru 0x018da5 (0xdca bytes) 000247[0] 1788199444.728451 punch from 0x022e9b to 0x0247c5 (0x192b bytes) 000248[0] 1788199444.755248 punch from 0x0400ea to 0x041a38 (0x194f bytes) 000250[1] 1788199444.785480 trunc from 0x055e70 to 0x2be48a (0x26861b bytes) 000251[1] 1788199444.859571 mapwrite 0x5c2eaa thru 0x5c3ca6 (0xdfd bytes) 000252[0] 1788199444.986521 write 0x1191fb thru 0x11b1e7 (0x1fed bytes) 000255[0] 1788199445.085613 mapread 0x4f3993 thru 0x4f5562 (0x1bd0 bytes) 000256[0] 1788199445.155457 punch from 0x4bbf1d to 0x4bc21d (0x301 bytes) 000259[1] 1788199445.206883 punch from 0x085482 to 0x0866b5 (0x1234 bytes) 000260[1] 1788199445.284102 read 0x173b5b thru 0x174d62 (0x1208 bytes) 000263[1] 1788199445.398797 mapwrite 0x48406e thru 0x484f83 (0xf16 bytes) 000264[0] 1788199445.541150 trunc from 0x5c3ca7 to 0x1193cc (0x4aa8da bytes) 000268[1] 1788199445.649038 write 0x6bf7a6 thru 0x6bfce8 (0x543 bytes) 000270[1] 1788199445.709387 trunc from 0x6bfce9 to 0x93fd16 (0x28002e bytes) 000275[0] 1788199445.884341 read 0x922185 thru 0x92224a (0xc6 bytes) 000276[0] 1788199445.910686 mapread 0x8aa94c thru 0x8ab576 (0xc2b bytes) 000278[1] 1788199445.949374 punch from 0x474bf9 to 0x476036 (0x143e bytes) 000279[1] 1788199445.956726 trunc from 0x93fd17 to 0x2da07e (0x665c98 bytes) 000280[0] 1788199446.021418 mapread 0x29cb74 thru 0x29ce9e (0x32b bytes) 000281[0] 1788199446.106561 mapwrite 0x924b78 thru 0x92581d (0xca6 bytes) 000282[1] 1788199446.313946 mapread 0x28437f thru 0x2856d0 (0x1352 bytes) 000283[1] 1788199446.355325 trunc from 0x92581e to 0x0a9e5d (0x87b9c0 bytes) 000284[0] 1788199446.406077 punch from 0x0106b0 to 0x010a72 (0x3c3 bytes) 000285[0] 1788199446.450894 punch from 0x031eea to 0x0337a7 (0x18be bytes) 000287[1] 1788199446.539597 punch from 0x05f7ab to 0x0616de (0x1f34 bytes) 000289[1] 1788199446.577771 write 0x951066 thru 0x952a9c (0x1a37 bytes) 000296[0] 1788199446.791356 mapread 0x7c6bce thru 0x7c863b (0x1a6e bytes) 000299[0] 1788199446.951479 mapwrite 0x4b1a25 thru 0x4b351d (0x1af9 bytes) 000300[0] 1788199447.230314 trunc from 0x952a9d to 0x09937d (0x8b971f bytes) 000302[1] 1788199447.331998 trunc from 0x09937e to 0x71c803 (0x683486 bytes) 000304[1] 1788199447.563910 mapwrite 0x239204 thru 0x23a578 (0x1375 bytes) 000307[0] 1788199447.800399 punch from 0x2e8410 to 0x2ea18c (0x1d7d bytes) 000310[1] 1788199447.927551 write 0x535eb6 thru 0x53622f (0x37a bytes) 000311[1] 1788199447.981143 mapread 0x2632f6 thru 0x26503f (0x1d4a bytes) 000312[1] 1788199448.102345 punch from 0x3968d3 to 0x3981ad (0x18db bytes) 000314[0] 1788199448.154757 read 0x0d402a thru 0x0d4e04 (0xddb bytes) 000315[0] 1788199448.195470 punch from 0x23bb02 to 0x23d94f (0x1e4e bytes) 000316[1] 1788199448.272559 punch from 0x573120 to 0x574be6 (0x1ac7 bytes) 000317[1] 1788199448.312233 trunc from 0x71c804 to 0x4e5fb4 (0x23684f bytes) 000318[0] 1788199448.357359 mapread 0x45f351 thru 0x46016d (0xe1d bytes) 000321[1] 1788199448.497314 mapread 0x0286e5 thru 0x0292ab (0xbc7 bytes) 000323[1] 1788199448.545302 punch from 0x203f29 to 0x20410f (0x1e7 bytes) 000327[0] 1788199448.627302 mapread 0x14703e thru 0x147694 (0x657 bytes) 000328[0] 1788199448.713280 mapwrite 0x63f9f8 thru 0x640bfe (0x1207 bytes) 000330[0] 1788199449.097035 mapwrite 0x423dce thru 0x424559 (0x78c bytes) 000333[0] 1788199449.285940 trunc from 0x640bff to 0x53ad76 (0x105e88 bytes) 000334[1] 1788199449.368260 punch from 0x44fd38 to 0x45181b (0x1ae4 bytes) 000337[0] 1788199449.503584 mapwrite 0x29601d thru 0x2966d1 (0x6b5 bytes) 000338[0] 1788199449.669770 write 0x7a4efd thru 0x7a6879 (0x197d bytes) 000339[0] 1788199449.735363 trunc from 0x7a687a to 0x79882d (0xe04c bytes) 000340[0] 1788199449.797093 mapread 0x509ac6 thru 0x50a803 (0xd3e bytes) 000341[1] 1788199449.883069 read 0x005257 thru 0x006375 (0x111f bytes) 000343[0] 1788199449.936077 punch from 0x76ab2a to 0x76b1d0 (0x6a7 bytes) 000344[1] 1788199449.946302 trunc from 0x79882e to 0x8a4bc4 (0x10c397 bytes) 000345[0] 1788199450.041168 write 0x4f2035 thru 0x4f33e5 (0x13b1 bytes) 000346[0] 1788199450.132691 punch from 0x75372e to 0x754763 (0x1036 bytes) 000347[0] 1788199450.184564 trunc from 0x8a4bc5 to 0x2df8ee (0x5c52d6 bytes) 000348[0] 1788199450.263090 mapwrite 0x744ded thru 0x744f7b (0x18f bytes) 000349[1] 1788199450.411651 trunc from 0x744f7c to 0x0e416e (0x660e0d bytes) 000350[1] 1788199450.485591 mapwrite 0x5697ce thru 0x56a16e (0x9a1 bytes) 000351[0] 1788199450.614098 trunc from 0x56a16f to 0x2a97d1 (0x2c099d bytes) truncating to largest ever: 0x9e0b0d 000352[0] 1788199450.666097 trunc from 0x2a97d2 to 0x9e0b0c (0x73733b bytes) 000353[1] 1788199450.696741 read 0x6fced1 thru 0x6fd4ca (0x5fa bytes) 000354[1] 1788199450.778447 mapread 0x54f6e2 thru 0x54f84d (0x16c bytes) 000355[0] 1788199450.828677 mapread 0x798d26 thru 0x79a032 (0x130d bytes) 000356[1] 1788199450.887440 mapread 0x4afdf6 thru 0x4b18ff (0x1b0a bytes) 000357[1] 1788199450.928608 punch from 0x275498 to 0x2771b7 (0x1d20 bytes) 000358[1] 1788199450.963335 punch from 0x14fe86 to 0x151aa2 (0x1c1d bytes) 000359[0] 1788199450.994295 punch from 0x55edf0 to 0x55fc79 (0xe8a bytes) 000361[0] 1788199451.103581 read 0x664292 thru 0x664357 (0xc6 bytes) 000362[1] 1788199451.136267 mapwrite 0x775c83 thru 0x77784e (0x1bcc bytes) 000363[1] 1788199451.404979 mapwrite 0x52486f thru 0x5259f9 (0x118b bytes) 000364[1] 1788199451.759862 punch from 0x972593 to 0x973f56 (0x19c4 bytes) 000365[0] 1788199451.823112 write 0x96d11b thru 0x96d251 (0x137 bytes) 000366[0] 1788199451.892480 write 0x9ecbb6 thru 0x9ee568 (0x19b3 bytes) 000369[1] 1788199451.976407 trunc from 0x9ee569 to 0x4adc9f (0x5408c9 bytes) 000370[1] 1788199452.027763 write 0x0a5beb thru 0x0a779a (0x1bb0 bytes) 000371[0] 1788199452.076425 read 0x3ae527 thru 0x3afc9f (0x1779 bytes) 000376[0] 1788199452.185529 write 0x288aec thru 0x28a513 (0x1a28 bytes) 000377[1] 1788199452.229582 mapwrite 0x0dc34c thru 0x0dd9d5 (0x168a bytes) 000380[1] 1788199452.432458 trunc from 0x4adca0 to 0x88ff5f (0x3e22c0 bytes) 000381[1] 1788199452.550002 trunc from 0x88ff60 to 0x6afefb (0x1e0064 bytes) 000382[1] 1788199452.602200 mapwrite 0x4a6685 thru 0x4a6803 (0x17f bytes) 000383[0] 1788199452.742907 mapread 0x32064e thru 0x320d75 (0x728 bytes) 000384[1] 1788199452.815938 trunc from 0x6afefc to 0x4b3f2c (0x1fbfcf bytes) 000385[1] 1788199452.890334 mapwrite 0x4b46f1 thru 0x4b6616 (0x1f26 bytes) 000387[0] 1788199453.060494 write 0x7163ab thru 0x71816e (0x1dc4 bytes) 000388[0] 1788199453.133566 mapread 0x3a3396 thru 0x3a3df4 (0xa5f bytes) 000389[1] 1788199453.160143 write 0x60e0f8 thru 0x60f0f0 (0xff9 bytes) 000392[1] 1788199453.205611 trunc from 0x71816f to 0x5a866d (0x16fb01 bytes) 000394[0] 1788199453.312658 punch from 0x5a2b5d to 0x5a2d6e (0x212 bytes) 000396[0] 1788199453.399213 trunc from 0x5a866e to 0x04ce03 (0x55b86a bytes) 000397[1] 1788199453.486003 mapread 0x021591 thru 0x021ac2 (0x532 bytes) 000398[1] 1788199453.551218 trunc from 0x04ce04 to 0x0eea09 (0xa1c06 bytes) 000399[0] 1788199453.671550 mapwrite 0x2fc47e thru 0x2fcd72 (0x8f5 bytes) 000400[1] 1788199453.886145 mapwrite 0x4f88e5 thru 0x4f9766 (0xe82 bytes) 000402[1] 1788199454.100570 mapread 0x364118 thru 0x364a20 (0x909 bytes) 000403[1] 1788199454.133173 punch from 0x40dd5c to 0x40fb2f (0x1dd4 bytes) 000404[0] 1788199454.142824 mapread 0x18f6e6 thru 0x18fc11 (0x52c bytes) 000405[0] 1788199454.173836 mapwrite 0x792e42 thru 0x793907 (0xac6 bytes) 000406[1] 1788199454.304415 trunc from 0x793908 to 0x0df073 (0x6b4894 bytes) 000407[1] 1788199454.369488 mapread 0x0dbee8 thru 0x0dca89 (0xba2 bytes) 000409[0] 1788199454.422970 punch from 0x0984e6 to 0x0989b8 (0x4d3 bytes) 000413[1] 1788199454.522589 trunc from 0x0df074 to 0x5ec71e (0x50d6ab bytes) 000414[1] 1788199454.584081 read 0x09dc28 thru 0x09f3ab (0x1784 bytes) 000415[1] 1788199454.607449 trunc from 0x5ec71f to 0x79719a (0x1aaa7c bytes) 000416[0] 1788199454.628191 trunc from 0x79719b to 0x0e24a7 (0x6b4cf3 bytes) 000417[1] 1788199454.694856 mapread 0x01b4b0 thru 0x01d455 (0x1fa6 bytes) 000418[0] 1788199454.746198 trunc from 0x0e24a8 to 0x49c83f (0x3ba398 bytes) 000419[1] 1788199454.777521 punch from 0x40f88d to 0x4115ef (0x1d63 bytes) 000420[1] 1788199454.814553 read 0x108f4a thru 0x109228 (0x2df bytes) 000421[0] 1788199454.844826 trunc from 0x49c840 to 0x71ec19 (0x2823da bytes) 000422[0] 1788199454.895135 mapwrite 0x5d6bf3 thru 0x5d872a (0x1b38 bytes) 000424[1] 1788199454.997576 mapread 0x20a3d7 thru 0x20a4e8 (0x112 bytes) 000427[0] 1788199455.075057 punch from 0x12134c to 0x121740 (0x3f5 bytes) 000428[1] 1788199455.092449 punch from 0x006d71 to 0x007880 (0xb10 bytes) 000430[1] 1788199455.130556 mapwrite 0x1066b8 thru 0x107e8e (0x17d7 bytes) 000434[1] 1788199455.250556 read 0x6c1019 thru 0x6c1313 (0x2fb bytes) 000436[1] 1788199455.287275 mapread 0x2d6ae8 thru 0x2d8759 (0x1c72 bytes) 000441[1] 1788199455.351968 mapwrite 0x4c4616 thru 0x4c61e9 (0x1bd4 bytes) 000442[0] 1788199455.502230 read 0x5ac299 thru 0x5ad329 (0x1091 bytes) 000443[0] 1788199455.575718 write 0x9226a6 thru 0x923dee (0x1749 bytes) 000444[1] 1788199455.602106 read 0x0a96e2 thru 0x0aa44d (0xd6c bytes) 000445[1] 1788199455.644919 read 0x4cd802 thru 0x4cf456 (0x1c55 bytes) 000447[0] 1788199455.655614 mapwrite 0x9e80f3 thru 0x9e9395 (0x12a3 bytes) 000449[1] 1788199455.783947 mapwrite 0x64fbba thru 0x6517a1 (0x1be8 bytes) 000450[1] 1788199455.936567 read 0x6ee737 thru 0x6ee775 (0x3f bytes) 000451[0] 1788199455.962201 mapwrite 0x22cf62 thru 0x22d297 (0x336 bytes) 000452[1] 1788199456.072801 mapwrite 0x259b6b thru 0x25aca5 (0x113b bytes) 000455[1] 1788199456.239741 punch from 0x555c22 to 0x55624c (0x62b bytes) 000456[1] 1788199456.274459 trunc from 0x9e9396 to 0x37e782 (0x66ac13 bytes) 000457[1] 1788199456.314430 trunc from 0x37e783 to 0x7f90e6 (0x47a964 bytes) 000458[0] 1788199456.358799 trunc from 0x7f90e7 to 0x459e09 (0x39f2dd bytes) 000459[1] 1788199456.416950 mapread 0x175bf8 thru 0x175d54 (0x15d bytes) 000460[0] 1788199456.461597 trunc from 0x459e0a to 0x089205 (0x3d0c04 bytes) 000461[0] 1788199456.558218 mapwrite 0x7bb700 thru 0x7bbfd3 (0x8d4 bytes) 000462[0] 1788199456.740297 mapwrite 0x1ab71d thru 0x1ac01f (0x903 bytes) 000463[1] 1788199456.820016 trunc from 0x7bbfd4 to 0x6867be (0x135815 bytes) 000464[0] 1788199456.877188 mapwrite 0x32b7ae thru 0x32ce0f (0x1662 bytes) 000465[1] 1788199457.011829 write 0x207dd2 thru 0x20935d (0x158c bytes) 000466[0] 1788199457.042272 mapread 0x5a68b9 thru 0x5a77e7 (0xf2f bytes) 000470[0] 1788199457.145549 mapwrite 0x78d166 thru 0x78d37a (0x215 bytes) 000473[1] 1788199457.320335 write 0x6c2039 thru 0x6c31f9 (0x11c1 bytes) 000474[1] 1788199457.392149 punch from 0x5f311a to 0x5f4fe7 (0x1ece bytes) 000477[1] 1788199457.450719 punch from 0x49f007 to 0x49ff05 (0xeff bytes) 000478[1] 1788199457.483058 read 0x215dfa thru 0x216644 (0x84b bytes) 000479[1] 1788199457.503549 trunc from 0x78d37b to 0x56e333 (0x21f047 bytes) 000481[1] 1788199457.556761 mapwrite 0x5746b1 thru 0x5751aa (0xafa bytes) 000483[0] 1788199457.636849 punch from 0x4336b4 to 0x434915 (0x1262 bytes) 000484[0] 1788199457.686084 write 0x0ea1a5 thru 0x0eb92c (0x1788 bytes) 000487[0] 1788199457.742114 mapread 0x3a767b thru 0x3a8840 (0x11c6 bytes) 000489[0] 1788199457.753114 punch from 0x296bc0 to 0x297bae (0xfef bytes) 000491[1] 1788199457.779953 punch from 0x4646f0 to 0x464b64 (0x475 bytes) 000492[1] 1788199457.811735 write 0x3cc11e thru 0x3cd57c (0x145f bytes) 000494[1] 1788199457.865843 punch from 0x4eedd7 to 0x4ef811 (0xa3b bytes) 000495[1] 1788199457.897889 trunc from 0x5751ab to 0x8b2515 (0x33d36b bytes) 000496[0] 1788199457.943814 read 0x3c63ff thru 0x3c741b (0x101d bytes) 000497[0] 1788199457.974469 mapwrite 0x0802cd thru 0x081cfa (0x1a2e bytes) 000498[1] 1788199458.068771 mapread 0x63c428 thru 0x63e0a6 (0x1c7f bytes) 000500[0] 1788199458.163036 write 0x130aef thru 0x1313dc (0x8ee bytes) All operations completed A-OK! PASS 16b (36s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 14:04:25 (1788199465) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 14:04:30 (1788199470) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.335708 s, 25.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.375371 s, 22.3 MB/s ldlm.namespaces.MGC192.168.204.158@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff99d2c34e9000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff99d2d2ced800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff99d2c34e9000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff99d2d2ced800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff99d2c34e9000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff99d2d2ced800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 15.4917 s, 6.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.21689 s, 24.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.7501 s, 38.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.71677 s, 18.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.16408 s, 12.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.03191 s, 26.0 MB/s PASS 16d (55s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 14:05:25 (1788199525) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.317772 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239082 s, 4.4 MB/s PASS 16e (7s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 14:05:32 (1788199532) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 14:06:00 (1788199560) TIMEOUT PASS 16g (29s) == sanityn test 16h: mmap read after truncate file ======= 14:06:29 (1788199589) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (7s) == sanityn test 16i: read after truncate file ============ 14:06:36 (1788199596) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (7s) == sanityn test 16j: race dio with buffered i/o ========== 14:06:44 (1788199604) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.17524 s, 12.1 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.37587 s, 19.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.65587 s, 9.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.57095 s, 16.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.11143 s, 12.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.40309 s, 10.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.14142 s, 8.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.83556 s, 14.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.40602 s, 10.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.22065 s, 6.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.40664 s, 5.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.26573 s, 8.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.64179 s, 7.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.07127 s, 6.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.43959 s, 5.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.11584 s, 6.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.85805 s, 5.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.47735 s, 4.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.861 s, 4.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (47s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 14:07:31 (1788199651) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1a7062c skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Mon Aug 31 14:08:28 EDT 2026 + fsx 2 done: rc=0 Mon Aug 31 14:08:28 EDT 2026 + fsx 3 done: rc=0 Mon Aug 31 14:08:28 EDT 2026 + fsx 4 done: rc=0 Mon Aug 31 14:08:28 EDT 2026 + fsx 5 done: rc=0 Mon Aug 31 14:08:28 EDT 2026 PASS 16k (68s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 14:08:40 (1788199720) fail_loc=0x8000030a PASS 17 (12s) == sanityn test 18: mmap sanity check =========================================================================================== 14:08:52 (1788199732) mmap test1: basic mmap operation (PASS, 0.28742s) mmap test2: MAP_PRIVATE not write back (PASS, 0.15093s) mmap test3: concurrent mmap ops on two nodes (PASS, 15.455s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.8025s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.30544s) mmap test6: check mmap write/read content on two nodes (PASS, 0.14954s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 2.0906s) mmap test11: multi-thread shared mmap (PASS, 1.924s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1023s) PASS 18 (36s) == sanityn test 19: test concurrent uncached read races ========================================================================= 14:09:28 (1788199768) oleg458-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg458-client: oleg458-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 14:09:33 (1788199773) PASS 20 (9s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 14:09:43 (1788199783) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (9s) == sanityn test 23: others should see updated atime while another read============================================================== 14:09:52 (1788199792) now is 1788199796 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 new atime is 1788199857 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 14:11:02 (1788199862) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 14:11:10 (1788199870) PASS 24b (7s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 14:11:17 (1788199877) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (6s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 14:11:23 (1788199883) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 14:11:27 (1788199887) utime: good mknod times 1788199888 <= 1788199889 <= 1788199889 for /mnt/lustre/f26a utime: good mknod times 1788199888 <= 1788199889 <= 1788199889 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (9s) == sanityn test 26b: sync mtime between ost and mds ====== 14:11:36 (1788199896) PASS 26b (11s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 14:11:47 (1788199907) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-08-31 14:11:49.000000000 -0400 Birth: 2026-08-31 14:11:49.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-08-31 14:11:49.000000000 -0400 Birth: 2026-08-31 14:11:49.000000000 -0400 PASS 26c (10s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 14:11:59 (1788199919) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 14:12:08 (1788199928) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 14:12:16 (1788199936) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg458-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.158@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (24s) == sanityn test 31r: open-rename(replace) race =========== 14:12:41 (1788199961) -rw-r--r-- 1 root root 0 Aug 31 14:12 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (11s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 14:12:52 (1788199972) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43208 Terminated tail -f $tfile PASS 31s (9s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 14:13:01 (1788199981) PASS 31t (10s) == sanityn test 32b: lockless i/o ======================== 14:13:11 (1788199991) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (5s) == sanityn test 32c: contention detection ================ 14:13:17 (1788199997) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.9766 s, 26.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.20503 s, 23.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.54647 s, 20.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.71502 s, 19.3 MB/s contention_seconds 4, contention_events 25 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.63451 s, 19.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.93559 s, 17.9 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.4315 s, 36.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.47017 s, 35.7 MB/s PASS 32c (37s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 14:13:57 (1788200037) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (5s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 14:14:02 (1788200042) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 14:14:07 (1788200047) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 14:14:11 (1788200051) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (5s) == sanityn test 34: no lock timeout under IO ============= 14:14:16 (1788200056) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg458-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff99d2c34e9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99d2c34e9000.ost_server_uuid in IDLE state after 0 sec oleg458-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99d2c34e9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99d2c34e9000.ost_server_uuid in FULL state after 0 sec oleg458-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff99d2c34e9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99d2c34e9000.ost_server_uuid in IDLE state after 0 sec oleg458-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99d2c34e9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99d2c34e9000.ost_server_uuid in FULL state after 0 sec oleg458-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff99d2c34e9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99d2c34e9000.ost_server_uuid in IDLE state after 0 sec oleg458-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99d2c34e9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99d2c34e9000.ost_server_uuid in FULL state after 0 sec PASS 34 (155s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 14:16:51 (1788200211) fail_loc=0x80000317 Race attempt 0 Wait for 50516 50533 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1855: 50516 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.158@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.204.158@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 - open/close 560 (time 1788200230.38 total 10.01 last 55.95) - open/close 1084 (time 1788200240.42 total 20.04 last 52.22) - open/close 1543 (time 1788200250.42 total 30.05 last 45.89) - open/close 2160 (time 1788200260.42 total 40.05 last 61.70) - open/close 2874 (time 1788200270.43 total 50.06 last 71.31) - open/close 3533 (time 1788200280.44 total 60.06 last 65.87) total: 4000 open/close in 69.42 seconds: 57.62 ops/second PASS 35 (78s) == sanityn test 36: handle ESTALE/open-unlink correctly == 14:18:10 (1788200290) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg458-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.807453 s, 64.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7440384) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.792875 s, 66.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.599808 s, 87.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7491584) after_dd(7427072) after(7478272) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.917665 s, 57.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7478272) after_dd(7437312) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.868476 s, 60.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7497728) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.810049 s, 64.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7492608) after_dd(7430144) after(7481344) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.912813 s, 57.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7481344) after_dd(7439360) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.08179 s, 48.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7497728) after_dd(7440384) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.05693 s, 49.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7497728) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.651349 s, 80.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7490560) after_dd(7442432) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.81169 s, 64.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(10) *** before(7497728) after_dd(7428096) after(7479296) PASS 36 (347s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 14:23:56 (1788200636) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 - create 1294 (time 1788200651.41 total 10.00 last 129.39) - create 2448 (time 1788200661.41 total 20.00 last 115.37) - create 3559 (time 1788200671.43 total 30.02 last 110.87) - create 4677 (time 1788200681.44 total 40.03 last 111.73) - create 5809 (time 1788200691.45 total 50.04 last 113.08) - create 6948 (time 1788200701.46 total 60.05 last 113.80) - create 8174 (time 1788200711.47 total 70.06 last 122.53) - create 9139 (time 1788200721.48 total 80.07 last 96.36) total: 10000 create in 88.80 seconds: 112.61 ops/second PASS 37 (104s) == sanityn test 39a: file mtime does not change after rename ========================================================== 14:25:40 (1788200740) repeat after cancel_lru_locks PASS 39a (10s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 14:25:51 (1788200751) repeat after cancel_lru_locks PASS 39b (11s) == sanityn test 39c: check truncate mtime update ================================================================================ 14:26:02 (1788200762) repeat after cancel_lru_locks PASS 39c (11s) == sanityn test 39d: sync write should update mtime ====== 14:26:13 (1788200773) fail_loc=0x411 fail_loc=0 PASS 39d (11s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 14:26:27 (1788200787) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (25s) == sanityn test 40c: pdirops: link and others ======================================================================== 14:26:52 (1788200812) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (27s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 14:27:19 (1788200839) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (22s) == sanityn test 40e: pdirops: rename and others ======================================================================== 14:27:41 (1788200861) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (24s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 14:28:06 (1788200886) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (22s) == sanityn test 41b: pdirops: create vs create ======================================================================== 14:28:27 (1788200907) open: File exists Conflict PASS 41b (17s) == sanityn test 41c: pdirops: create vs link ======================================================================== 14:28:44 (1788200924) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (15s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 14:28:59 (1788200939) Conflict PASS 41d (17s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 14:29:16 (1788200956) Conflict PASS 41e (17s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 14:29:34 (1788200974) Conflict PASS 41f (16s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 14:29:50 (1788200990) Conflict PASS 41g (19s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 14:30:08 (1788201008) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (18s) == sanityn test 41i: reint_open: create vs create ======== 14:30:26 (1788201026) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1173s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 14:49:59 (1788202199) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (15s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 14:50:14 (1788202214) open: File exists Conflict PASS 42b (13s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 14:50:28 (1788202228) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (13s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 14:50:42 (1788202242) Conflict PASS 42d (13s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 14:50:55 (1788202255) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (13s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 14:51:09 (1788202269) Conflict PASS 42f (14s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 14:51:23 (1788202283) Conflict PASS 42g (15s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 14:51:38 (1788202298) Conflict PASS 42h (14s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 14:51:52 (1788202312) PASS 43a (63s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 14:52:55 (1788202375) Conflict PASS 43b (14s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 14:53:10 (1788202390) Conflict PASS 43c (15s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 14:53:25 (1788202405) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (15s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 14:53:40 (1788202420) Conflict PASS 43e (15s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 14:53:55 (1788202435) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (14s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 14:54:10 (1788202450) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (16s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 14:54:26 (1788202466) Conflict PASS 43h (15s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 14:54:41 (1788202481) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 14:54:44 (1788202484) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (131s) == sanityn test 43k: unlink vs create ==================== 14:56:55 (1788202615) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1307s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 15:18:43 (1788203923) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Mon Aug 31 15:18:52 EDT 2026 PASS 44a (15s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 15:18:59 (1788203939) open: File exists Conflict PASS 44b (14s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 15:19:13 (1788203953) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (15s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 15:19:28 (1788203968) Conflict PASS 44d (15s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 15:19:43 (1788203983) Conflict PASS 44e (15s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 15:19:58 (1788203998) Conflict PASS 44f (13s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 15:20:11 (1788204011) Conflict PASS 44g (14s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 15:20:25 (1788204025) Conflict PASS 44h (17s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 15:20:42 (1788204042) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (5s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 15:20:47 (1788204047) PASS 45a (108s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 15:22:35 (1788204155) Conflict PASS 45b (15s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 15:22:51 (1788204171) Conflict PASS 45c (15s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 15:23:05 (1788204185) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (17s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 15:23:22 (1788204202) Conflict PASS 45e (17s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 15:23:39 (1788204219) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (18s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 15:23:57 (1788204237) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (15s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 15:24:12 (1788204252) Conflict PASS 45h (19s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 15:24:31 (1788204271) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (5s) == sanityn test 45j: read vs rename ====================== 15:24:36 (1788204276) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1323s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 15:46:39 (1788205599) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (14s) == sanityn test 46b: pdirops: link vs create ======================================================================== 15:46:54 (1788205614) open: File exists Conflict PASS 46b (13s) == sanityn test 46c: pdirops: link vs link ======================================================================== 15:47:07 (1788205627) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (14s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 15:47:20 (1788205640) Conflict PASS 46d (14s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 15:47:34 (1788205654) Conflict PASS 46e (13s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 15:47:47 (1788205667) Conflict PASS 46f (14s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 15:48:01 (1788205681) Conflict PASS 46g (15s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 15:48:16 (1788205696) Conflict PASS 46h (15s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 15:48:31 (1788205711) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (4s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 15:48:35 (1788205715) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 15:48:39 (1788205719) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (5s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 15:48:44 (1788205724) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (5s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 15:48:49 (1788205729) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (5s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 15:48:54 (1788205734) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (5s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 15:48:59 (1788205739) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (5s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 15:49:04 (1788205744) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (5s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 15:49:09 (1788205749) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.144097 s, 71.1 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (12s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 15:49:21 (1788205761) 0+1 records in 0+1 records out 158 bytes copied, 0.0369167 s, 4.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218839) - No such process PASS 51a (11s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 15:49:32 (1788205772) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0165129 s, 62.0 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111674 s, 9.4 MB/s 3145728 PASS 51b (21s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 15:49:53 (1788205793) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0238413 s, 43.0 kB/s PASS 51c (12s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 15:50:06 (1788205806) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540733 s, 19.4 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (9s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 15:50:15 (1788205815) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 read lease(1) has applied. PASS 51e (11s) == sanityn test 54: rename locking ======================= 15:50:26 (1788205826) llite.lustre-ffff99d2c34e9000.xattr_cache=1 llite.lustre-ffff99d2d2ced800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-31 15:50:28.000000000 -0400 Modify: 2026-08-31 15:50:28.000000000 -0400 Change: 2026-08-31 15:50:28.000000000 -0400 Birth: 2026-08-31 15:50:27.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-31 15:50:35.000000000 -0400 Modify: 2026-08-31 15:50:36.000000000 -0400 Change: 2026-08-31 15:50:36.000000000 -0400 Birth: 2026-08-31 15:50:34.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-31 15:50:41.000000000 -0400 Modify: 2026-08-31 15:50:41.000000000 -0400 Change: 2026-08-31 15:50:41.000000000 -0400 Birth: 2026-08-31 15:50:41.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-31 15:50:41.000000000 -0400 Modify: 2026-08-31 15:50:43.000000000 -0400 Change: 2026-08-31 15:50:43.000000000 -0400 Birth: 2026-08-31 15:50:41.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-31 15:50:48.000000000 -0400 Modify: 2026-08-31 15:50:50.000000000 -0400 Change: 2026-08-31 15:50:50.000000000 -0400 Birth: 2026-08-31 15:50:48.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-31 15:50:48.000000000 -0400 Modify: 2026-08-31 15:50:50.000000000 -0400 Change: 2026-08-31 15:50:50.000000000 -0400 Birth: 2026-08-31 15:50:48.000000000 -0400 PASS 54 (36s) == sanityn test 55a: rename vs unlink target dir ========= 15:51:02 (1788205862) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (16s) == sanityn test 55b: rename vs unlink source dir ========= 15:51:18 (1788205878) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (14s) == sanityn test 55c: rename vs unlink orphan target dir == 15:51:32 (1788205892) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (21s) == sanityn test 55d: rename file vs link ================= 15:51:53 (1788205913) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (17s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 15:52:10 (1788205930) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (4s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 15:52:14 (1788205934) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (20s) == sanityn test 55g: rename: race with trylock =========== 15:52:34 (1788205954) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (25s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 15:53:00 (1788205980) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788205982 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.701598 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (96s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 15:54:35 (1788206075) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788206081 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.15164 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (121s) == sanityn test 60: Verify data_version behaviour ======== 15:56:36 (1788206196) PASS 60 (8s) == sanityn test 70a: cd directory && rm directory ======== 15:56:44 (1788206204) PASS 70a (8s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 15:56:52 (1788206212) PASS 70b (10s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 15:57:02 (1788206222) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0216288 s, 3.0 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (5s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 15:57:07 (1788206227) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00889933 s, 7.4 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 15:57:10 (1788206230) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (4s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 15:57:14 (1788206234) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0209445 s, 3.1 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (4s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 15:57:18 (1788206238) llite.lustre-ffff99d2c34e9000.xattr_cache=1 llite.lustre-ffff99d2d2ced800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (8s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 15:57:26 (1788206246) llite.lustre-ffff99d2c34e9000.xattr_cache=1 llite.lustre-ffff99d2d2ced800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (9s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 15:57:35 (1788206255) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0193383 s, 53.0 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0334142 s, 30.6 kB/s 231659: get lock2 231660: get lock1 231660: done 231660: get lock2 231659: done 231659: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 231659: cannot get lock1: Resource deadlock avoided 231660: done 231660: exit rc=0 231659: done 231659: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 15:57:46 (1788206266) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0490955 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387364 s, 27.1 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232241 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (14s) == sanityn test 76: Verify MDT open_files listing ======== 15:58:00 (1788206280) opening files: ........ closing files: ........ PASS 76 (70s) == sanityn test 77a: check FIFO NRS policy =============== 15:59:10 (1788206350) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.308896 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156815 s, 66.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183967 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488645 s, 21.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675139 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204222 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103972 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794493 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056477 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379284 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231721 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18426 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160811 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19099 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200926 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284658 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99d2d2ced800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff99d2d2ced800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293716 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147897 s, 70.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101502 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977413 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654531 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715707 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124657 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118148 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09137 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505872 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510485 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979321 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111963 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673068 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106856 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967249 s, 10.8 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (11s) == sanityn test 77b: check CRR-N NRS policy ============== 15:59:21 (1788206361) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.303965 s, 55.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424299 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02565 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609316 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116098 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123459 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208322 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1036 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22309 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198851 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161989 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258166 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272391 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306615 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26182 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249119 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99d2d2ced800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff99d2d2ced800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333472 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349147 s, 30.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117781 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118104 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385691 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042814 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886859 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500276 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941365 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352087 s, 29.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809387 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993234 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384609 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103984 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715414 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676263 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557177 s, 18.8 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.614749 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549915 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908332 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486972 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307068 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602671 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613719 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808363 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507888 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214815 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176226 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287532 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252873 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32239 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312896 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315461 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176214 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294057 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197104 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688547 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434651 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612912 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426393 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774142 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040224 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449897 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603642 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057846 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542037 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984157 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710027 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790887 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913269 s, 11.5 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg458-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg458-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (17s) == sanityn test 77c: check ORR NRS policy ================ 15:59:38 (1788206378) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.396839 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312393 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689645 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857007 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705058 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1258 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324588 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840116 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114251 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923022 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271704 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2519 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23181 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221842 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212619 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184331 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374209 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757374 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00629561 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157826 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713925 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180797 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134165 s, 78.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119229 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130905 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160126 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15165 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149378 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922128 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595694 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102296 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491717 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059747 s, 17.6 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.186976 s, 89.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359997 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329545 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047278 s, 22.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588996 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404168 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079973 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378793 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14339 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10317 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981713 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101991 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123139 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107792 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186295 s, 5.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99d2d2ced800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff99d2d2ced800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253039 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203706 s, 5.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99d2d2ced800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469358 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304688 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091421 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00393104 s, 267 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00528149 s, 199 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00221377 s, 474 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00245154 s, 428 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115651 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528334 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584742 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492724 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498436 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407386 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040883 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363483 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351988 s, 29.8 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg458-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg458-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.574927 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902531 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694584 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204624 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232532 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134927 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189355 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209788 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251095 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136586 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222965 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224993 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197906 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302793 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275162 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23989 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99d2d2ced800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff99d2d2ced800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284839 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109301 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00275907 s, 380 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15911 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237314 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150034 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187105 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923786 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866078 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176249 s, 59.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783636 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931971 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516438 s, 20.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349162 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614038 s, 17.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533717 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767127 s, 13.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (20s) == sanityn test 77d: check TRR nrs policy ================ 15:59:58 (1788206398) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.291728 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776026 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543041 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919579 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108252 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17444 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176273 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161342 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19473 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198309 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147297 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173547 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222158 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197864 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217635 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25377 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99d2d2ced800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff99d2d2ced800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199598 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00237522 s, 441 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143801 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125748 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173014 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1077 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173353 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888193 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104947 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126751 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781642 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857346 s, 12.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795305 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732147 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693763 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831943 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914161 s, 11.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg458-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg458-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.482385 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631878 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524356 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244076 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898853 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094026 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722779 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212652 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162134 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257381 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704223 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267048 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275664 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226433 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272726 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2642 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99d2d2ced800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff99d2d2ced800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946679 s, 11.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff99d2d2ced800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00255901 s, 410 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113916 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160136 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224339 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119045 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207369 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186378 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470122 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425266 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150161 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11793 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986861 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099402 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198757 s, 5.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904187 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843024 s, 12.4 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg458-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg458-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.275838 s, 60.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542343 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543198 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063331 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768636 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176754 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952653 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17842 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472168 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219561 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128867 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138017 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339686 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29251 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373916 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23986 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99d2d2ced800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345437 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189967 s, 55.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113276 s, 92.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647719 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118746 s, 8.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff99d2d2ced800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00788478 s, 133 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10329 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100406 s, 104 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121305 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137275 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146524 s, 71.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758489 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257718 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632186 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609246 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510794 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476959 s, 22.0 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (18s) == sanityn test 77e: check TBF NID nrs policy ============ 16:00:16 (1788206416) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.58@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.182301 s, 92.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497739 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476377 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808952 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958626 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842195 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802648 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108757 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977922 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104661 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117295 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938635 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113284 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987939 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942868 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955989 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126011 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968222 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498695 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115508 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121171 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676149 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683588 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720891 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142138 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816539 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104243 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09603 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105658 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112754 s, 9.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143126 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184182 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147946 s, 7.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.317292 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401502 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645882 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959918 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093035 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164145 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186778 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240495 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267575 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230343 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222132 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231256 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241876 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2694 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289273 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326974 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99d2d2ced800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff99d2d2ced800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364446 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14553 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129147 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682648 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174809 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207915 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128702 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157024 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157571 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168239 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163931 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173424 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13333 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185768 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174744 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121304 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195461 s, 5.4 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.619266 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271805 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539179 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115627 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856008 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172422 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127888 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194816 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165566 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141715 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158962 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177009 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235051 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210741 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214587 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244601 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281689 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907804 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241077 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123182 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848751 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573127 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127275 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134991 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10666 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00392108 s, 267 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981197 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067361 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101214 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00398637 s, 263 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906418 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0062499 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490534 s, 21.4 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg458-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg458-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.359941 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387446 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398841 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103121 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168561 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119525 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177848 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234825 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142769 s, 7.3 MB/sldlm.namespaces.lustre-OST0001-osc-ffff99d2d2ced800.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219825 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198382 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222176 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240856 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302959 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323829 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278576 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420416 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299825 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121505 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571751 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00859911 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131442 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00228702 s, 458 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942674 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143212 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277091 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161071 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989017 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587896 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048789 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416027 s, 25.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036735 s, 28.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252554 s, 41.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg458-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg458-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (31s) == sanityn test 77f: check TBF JobID nrs policy ========== 16:00:47 (1788206447) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg458-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.40876 s, 41.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570514 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376298 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101519 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534442 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114447 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105071 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168245 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104435 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028345 s, 37.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99d2d2ced800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198227 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212698 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198918 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140418 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178859 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275209 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277785 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00524253 s, 200 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103184 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453395 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658969 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00731846 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332672 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364503 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455281 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00571994 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108707 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480345 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00470546 s, 223 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00671149 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00355051 s, 295 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0028117 s, 373 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134171 s, 78.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg458-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.34099 s, 49.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453081 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456015 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16727 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208349 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168739 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216181 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266462 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253609 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183174 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269403 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23489 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278249 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249948 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268388 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298393 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389893 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99d2d2ced800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff99d2d2ced800.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00215415 s, 487 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105221 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811739 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196837 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553325 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00984361 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104659 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166358 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1533 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161093 s, 6.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155616 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101873 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205138 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275789 s, 38.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174392 s, 60.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569173 s, 18.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg458-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.409758 s, 40.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05175 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700994 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052567 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10963 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736606 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153791 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111036 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159222 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119191 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135955 s, 7.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99d2d2ced800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312946 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206703 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206955 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236284 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292674 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383075 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00400162 s, 262 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00228042 s, 460 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125019 s, 83.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0071482 s, 147 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00513348 s, 204 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122584 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556316 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501882 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108944 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103291 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753156 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303959 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428457 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115825 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703765 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812721 s, 12.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg458-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.368934 s, 45.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249051 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416846 s, 25.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12848 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747421 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114603 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153179 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163324 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135704 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153496 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174982 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148026 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156396 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18587 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231838 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193865 s, 5.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99d2d2ced800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff99d2d2ced800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329461 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256974 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268981 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195064 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199524 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234378 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182155 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314611 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140224 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037675 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195139 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121849 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117161 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157667 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966892 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113605 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152093 s, 6.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg458-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg458-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (28s) == sanityn test 77g: Change TBF type directly ============ 16:01:16 (1788206476) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (15s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 16:01:30 (1788206490) oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg458-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg458-client: oleg458-server: ssh exited with exit code 2 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg458-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg458-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg458-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 PASS 77h (14s) == sanityn test 77i: Change rank of TBF rule ============= 16:01:44 (1788206504) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (27s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 16:02:11 (1788206531) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.488689 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314368 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031987 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353949 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816145 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724249 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130569 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111357 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111377 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101308 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167373 s, 6.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128488 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193312 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113729 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306434 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239971 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342368 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99d2d2ced800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00342644 s, 306 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15793 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00249252 s, 421 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939622 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153241 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106989 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0079328 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136445 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698686 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12107 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123794 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10098 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958839 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103074 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108259 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133769 s, 7.8 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg458-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg458-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.07194 s, 13.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6435 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (66s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 16:03:17 (1788206597) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg458-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg458-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg458-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg458-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg458-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.305 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854454 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999663 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15789 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149024 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231052 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292836 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320128 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29301 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340201 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3491 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351633 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343618 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296897 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45895 s, 2.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3618 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.560286 s, 1.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639106 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10178 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541074 s, 19.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519024 s, 20.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624379 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810919 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13418 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722998 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090357 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737161 s, 14.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929028 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100725 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986895 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107197 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121358 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161109 s, 6.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg458-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg458-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0271 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8516 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg458-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg458-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg458-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg458-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg458-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.25734 s, 13.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862329 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113126 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147632 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131527 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190251 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144634 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241246 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108748 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247391 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135283 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19976 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157337 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262987 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307996 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259854 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316777 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757617 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849226 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814091 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056902 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830654 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118708 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127048 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135746 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148899 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137598 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894362 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127614 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126511 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176826 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107652 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13117 s, 8.0 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.2058 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9394 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (157s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 16:05:55 (1788206755) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg458-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg458-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg458-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg458-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg458-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27501 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110692 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300547 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148238 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159602 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133559 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478083 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793811 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847816 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293348 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154125 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303769 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111551 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803388 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37271 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178434 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302656 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886422 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943424 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096781 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970942 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825868 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122951 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512987 s, 20.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959667 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893426 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11858 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877228 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854669 s, 12.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109615 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112024 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108677 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124246 s, 8.4 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9556 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9601 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg458-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg458-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg458-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg458-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg458-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27496 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875712 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918779 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633403 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997117 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2482 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162163 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239135 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190592 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237034 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209379 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327696 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285009 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31393 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252481 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379263 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.440623 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599333 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419528 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031713 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679213 s, 15.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048453 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858209 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507339 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754258 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065081 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592183 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108889 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1143 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820342 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122796 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752082 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108143 s, 9.7 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1433 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9593 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (162s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 16:08:36 (1788206916) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg458-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.530176 s, 31.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504043 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031766 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770954 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128857 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121837 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111085 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202151 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180991 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159276 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186976 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168679 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242553 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248688 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204953 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245667 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27673 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344885 s, 30.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081357 s, 12.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752948 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390856 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091196 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434305 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833911 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594713 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049439 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550469 s, 19.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105033 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670951 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694231 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673118 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116575 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846424 s, 12.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.45298 s, 12.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93056 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.58@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.58@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.453731 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628838 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806542 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189394 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164516 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213872 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195617 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227333 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226875 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274734 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263719 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253255 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260284 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320781 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27655 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341879 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382983 s, 2.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041494 s, 25.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100769 s, 104 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555719 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126078 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115284 s, 91.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999973 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101275 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989583 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013152 s, 79.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113261 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866144 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106786 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104313 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103914 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105741 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110735 s, 9.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.66859 s, 13.7 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98482 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.58@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg458-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.291833 s, 57.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425332 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907627 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13519 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151607 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915304 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873346 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128589 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131058 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128273 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127814 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976346 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168538 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155726 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186635 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159668 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152216 s, 6.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05532 s, 19.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653937 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092049 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129991 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738406 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063638 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070614 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122405 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12248 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100483 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084164 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090897 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921263 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103462 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14245 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128412 s, 8.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg458-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg458-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.97565 s, 11.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.22049 s, 20.1 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg458-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.29149 s, 57.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335233 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364604 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439666 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802832 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170386 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114879 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164432 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195014 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978456 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140297 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208424 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12532 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213536 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157708 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196428 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209874 s, 5.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105301 s, 99.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045061 s, 23.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168997 s, 62.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536238 s, 19.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796861 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307966 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373409 s, 28.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547752 s, 19.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037507 s, 28.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413781 s, 25.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433072 s, 24.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459785 s, 22.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453429 s, 23.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537218 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563429 s, 18.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536996 s, 19.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg458-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg458-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.2832 s, 8.5 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0681 s, 10.4 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9729 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0485 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg458-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.04387 s, 16.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453919 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240109 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481739 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100178 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106629 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103748 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600519 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217831 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163365 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235499 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149937 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228127 s, 4.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230657 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150287 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205662 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260941 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475729 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609595 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893812 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571128 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688693 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957246 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906956 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704876 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813024 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863121 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737089 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722492 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925997 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897493 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894388 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970066 s, 10.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg458-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg458-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9367 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6492 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg458-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.438114 s, 38.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448791 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544382 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589621 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866663 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122815 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116949 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289461 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510149 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156881 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161406 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157198 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286759 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279294 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297487 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295718 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271442 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540162 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123058 s, 85.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248335 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106525 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120487 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657058 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131722 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707357 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00932736 s, 112 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709715 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1354 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639052 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162334 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706432 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149524 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730122 s, 14.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg458-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg458-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.718 s, 12.0 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0532 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg458-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.631689 s, 26.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352126 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389975 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059998 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09285 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296363 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549357 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623919 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15481 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18111 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109212 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916651 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186749 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124416 s, 8.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107814 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143621 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202653 s, 5.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886555 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880273 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113535 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115779 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939833 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105644 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111091 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12567 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140307 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100869 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156009 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10699 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153499 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135685 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167327 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16114 s, 6.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg458-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg458-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.345 s, 9.2 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92355 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.5117 s, 9.1 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95882 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (448s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 16:16:04 (1788207364) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg458-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.59958 s, 28.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813427 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874516 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132973 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168564 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138109 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17324 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963344 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12067 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243252 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28279 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342025 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298895 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248914 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330771 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347444 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293285 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99d2d2ced800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff99d2d2ced800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367222 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237102 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046395 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140551 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050406 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415944 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145855 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146188 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640171 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669063 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701299 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887898 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814013 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914522 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989402 s, 10.6 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg458-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg458-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.0981 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95738 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (57s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 16:17:01 (1788207421) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.58@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.58@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.317734 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475348 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337765 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510564 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192968 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039181 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374803 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299637 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215276 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554545 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159089 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152096 s, 6.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170445 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185236 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914339 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942288 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251603 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698236 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555244 s, 18.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101812 s, 103 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451172 s, 23.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050071 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423569 s, 24.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484245 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707706 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614186 s, 17.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119946 s, 87.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770026 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00556461 s, 188 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830387 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479359 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093034 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949502 s, 11.0 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.49402 s, 14.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93361 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (54s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 16:17:55 (1788207475) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.58@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg458-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.328368 s, 51.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876576 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998086 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14971 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126665 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123392 s, 8.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105063 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194214 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188017 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216245 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163306 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141931 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201082 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24475 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278161 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269302 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24443 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769155 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601348 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786624 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890229 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507141 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061307 s, 17.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624105 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065419 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825326 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102278 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900213 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080701 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901315 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113127 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987314 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119791 s, 8.8 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.98411 s, 11.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.21994 s, 20.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (49s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 16:18:44 (1788207524) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg458-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.607788 s, 27.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377719 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531456 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443469 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02442 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533555 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476833 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1023 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102654 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079516 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665421 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160806 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143351 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363652 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284476 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.414993 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.439927 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439912 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610835 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025139 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996876 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870564 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479119 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080671 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823208 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981012 s, 10.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622968 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11863 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823241 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109273 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776448 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809055 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859095 s, 12.2 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1652 s, 9.4 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97372 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.15987 s, 11.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93947 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (99s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 16:20:23 (1788207623) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg458-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.01795 s, 16.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811653 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639073 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840984 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138964 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194265 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619029 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203602 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185305 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138422 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186234 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175683 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249892 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218843 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15346 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185388 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252891 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750425 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853221 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131963 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892889 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849259 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863818 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106706 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842588 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111211 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111899 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106678 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116112 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997737 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109237 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125243 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125542 s, 8.4 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg458-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg458-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8913 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0199 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (74s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 16:21:38 (1788207698) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg458-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.67871 s, 24.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878446 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927238 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131124 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122975 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148565 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10365 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18589 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168001 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162023 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215489 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191126 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254207 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250706 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227769 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201417 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190961 s, 5.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99d2d2ced800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff99d2d2ced800.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705157 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633749 s, 16.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106449 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997671 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786603 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974233 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999616 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107823 s, 9.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137936 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100411 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973844 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128401 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105226 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139221 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120126 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127351 s, 8.2 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.06208 s, 11.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97025 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg458-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.384341 s, 43.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207411 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176627 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496614 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343183 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149647 s, 70.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810274 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044848 s, 23.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165756 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783148 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139767 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137071 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147984 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30212 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285287 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293015 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343112 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617933 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668252 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571371 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602214 s, 17.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646425 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695519 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110893 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547924 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133409 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823854 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884047 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897755 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955645 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966523 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108519 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139006 s, 7.5 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7882 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0182 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.5726 s, 9.1 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94097 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (144s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 16:24:02 (1788207842) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg458-client.virtnet /mnt/lustre (opts:) Stopping client oleg458-client.virtnet /mnt/lustre opts: Stopping clients: oleg458-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg458-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg458-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg458-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg458-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg458-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.58,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg458-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.58,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg458-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.58,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69217/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.58,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg458-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg458-server' oleg458-server: oleg458-server.virtnet: executing load_modules_local oleg458-server: kmemleak disabled oleg458-server: Loading modules from /home/green/git/lustre-release/lustre oleg458-server: detected 4 online CPUs by sysfs oleg458-server: MODOPTS_LIBCFS= oleg458-server: Force libcfs to create 2 CPU partitions oleg458-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg458-server: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg458-server: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg458-server: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre Starting client oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre Started clients oleg458-client.virtnet: 192.168.204.158@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre2 Starting client oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre2 Started clients oleg458-client.virtnet: 192.168.204.158@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff99d2c481a800.idle_timeout=debug osc.lustre-OST0000-osc-ffff99d2d0cf1800.idle_timeout=debug osc.lustre-OST0001-osc-ffff99d2c481a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff99d2d0cf1800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.44271 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113305 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101883 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134039 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1616 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146119 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18929 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320037 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313564 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299738 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313089 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289961 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370588 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318198 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369686 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3308 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319373 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99d2d0cf1800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff99d2d0cf1800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378455 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421584 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396683 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.438006 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403766 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408486 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.427315 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419618 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.436592 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426819 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429978 s, 2.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.424345 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.449063 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.458852 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45785 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.463795 s, 2.3 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.45358 s, 12.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.44555 s, 19.3 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.640023 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480398 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013971 s, 75.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887676 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640858 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112509 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110713 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146376 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824004 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426429 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107849 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184509 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156904 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185351 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19039 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143117 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167956 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423895 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10833 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543857 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09975 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967917 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778609 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1286 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112909 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918873 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118999 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118612 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142314 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121375 s, 8.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146993 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125878 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144618 s, 7.3 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg458-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg458-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2075 s, 10.3 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99714 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (249s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 16:28:11 (1788208091) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg458-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (22s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 16:28:33 (1788208113) On MGS 192.168.204.158, active = nodemap.active=1 On MGS 192.168.204.158, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.760668 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979577 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842095 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229553 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275704 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202366 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270102 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232141 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367659 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402503 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364316 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354202 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416585 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.460683 s, 2.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.494572 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.439029 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.489173 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880844 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114546 s, 9.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382228 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317437 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812494 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727185 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352111 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997209 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111559 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846462 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108652 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640869 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861896 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847897 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136689 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997263 s, 10.5 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.48838 s, 11.1 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.70876 s, 18.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.204.158, TBF.id = On MGS 192.168.204.158, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.618416 s, 27.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719659 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659671 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988464 s, 10.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090118 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104385 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220698 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192915 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119489 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150176 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258912 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151143 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253595 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134683 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186678 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239545 s, 4.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff99d2d0cf1800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246475 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483205 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102893 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102187 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041204 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204542 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378386 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806822 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464153 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729168 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761394 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831057 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00699734 s, 150 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894883 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729156 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097521 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00764615 s, 137 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.21226 s, 11.4 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.5895 s, 40.5 MB/s Read runtime is 3 s, speed is 33.333333 IOPS read rate (33.333333) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg458-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg458-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 On MGS 192.168.204.158, active = nodemap.active=1 On MGS 192.168.204.158, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.654889 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918756 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105582 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122725 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223026 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188104 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242232 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256311 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273875 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341483 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217902 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25471 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245464 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36914 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255211 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328683 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285303 s, 3.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff99d2d0cf1800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525657 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801404 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903007 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109584 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133719 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763176 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649148 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740634 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114497 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100393 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112527 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184255 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114568 s, 9.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122476 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128426 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143007 s, 7.3 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg458-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg458-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.34282 s, 11.2 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0558 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.204.158, TBF.id = On MGS 192.168.204.158, active = nodemap.active=0 PASS 77kj (193s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 16:31:46 (1788208306) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (13s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 16:31:59 (1788208319) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg458-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.25736 s, 322 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg458-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0808 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (58s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 16:32:57 (1788208377) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.296091 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103106 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837433 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101818 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093843 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136613 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992766 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26454 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239033 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264754 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358504 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379553 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340627 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398988 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39603 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366242 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233873 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566446 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00413237 s, 254 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109287 s, 95.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471645 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00370748 s, 283 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115585 s, 90.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00611993 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119179 s, 88.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11589 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00478679 s, 219 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608709 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143182 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104263 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255396 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208315 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206291 s, 5.1 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg458-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg458-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.93072 s, 11.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.27681 s, 19.9 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.579711 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991041 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111199 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123056 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143537 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150383 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160687 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128988 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21131 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15327 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187253 s, 5.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123173 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199481 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225236 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265807 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229436 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25178 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99d2d0cf1800.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482838 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556474 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361002 s, 29.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958182 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505344 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813106 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840401 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976281 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882034 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104932 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173536 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111213 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996232 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114193 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700458 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186325 s, 5.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99d2d0cf1800.lock_unused_count=2 removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg458-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg458-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.87674 s, 11.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.2235 s, 20.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (91s) == sanityn test 77o: Changing rank should not panic ====== 16:34:28 (1788208468) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (13s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 16:34:41 (1788208481) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (113s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 16:36:34 (1788208594) mds.MDS.mdt.nrs_policies=tbf oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg458-client: oleg458-server: ssh exited with exit code 36 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 oleg458-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg458-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (36s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 16:37:10 (1788208630) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273286 kill md_thread xargs PASS 77r (53s) == sanityn test 77s: Check TBF LRU shrinker ============== 16:38:04 (1788208684) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (63s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 16:39:07 (1788208747) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (9s) == sanityn test 79: xattr: intent error ================== 16:39:17 (1788208757) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935685 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-31 16:39:19.000000000 -0400 Modify: 2026-08-31 16:39:19.000000000 -0400 Change: 2026-08-31 16:39:19.000000000 -0400 Birth: 2026-08-31 16:39:18.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (21s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 16:39:37 (1788208777) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 16:39:40 (1788208780) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (4s) == sanityn test 81a: rename and stat under striped directory ========================================================== 16:39:44 (1788208784) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 16:39:47 (1788208787) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 16:39:50 (1788208790) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 16:39:54 (1788208794) - open/close 752 (time 1788208809.88 total 10.02 last 75.08) - open/close 1483 (time 1788208819.89 total 20.02 last 73.04) - open/close 2241 (time 1788208829.90 total 30.04 last 75.71) total: 2500 open/close in 33.47 seconds: 74.69 ops/second snapshot_time 1788208920.524486094 secs.nsecs start_time 1788208796.729101649 secs.nsecs elapsed_time 123.795384445 secs.nsecs open 2505 samples [usecs] 154 38341 4085467 12751017379 close 2504 samples [usecs] 179 7661 1011872 810705164 mknod 2501 samples [usecs] 591 38210 3627250 10897805226 unlink 2500 samples [usecs] 712 118097 5837812 60768487018 mkdir 2 samples [usecs] 1016 12391 13407 154569137 rmdir 2 samples [usecs] 9860 10026 19886 197740276 rename 2500 samples [usecs] 512 35973 14991778 170478245912 getattr 3762 samples [usecs] 10 9211 269630 414199586 parallel_rename_file 2500 samples [usecs] 512 35973 14991778 170478245912 crossdir_rename 2500 samples [usecs] 512 35973 14991778 170478245912 readpage 18 samples [usecs] 5593 83341 504792 22961614268 PASS 81d (132s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 16:42:06 (1788208926) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7634 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7634 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 16:42:13 (1788208933) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 16:42:17 (1788208937) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 16:42:31 (1788208951) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788208953 Finishing test test1 at 1788208957 PASS 85 (11s) == sanityn test 90: open/create and unlink striped directory ========================================================== 16:42:42 (1788208962) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 16:42:45 (1788208965) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 16:42:48 (1788208968) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 16:42:52 (1788208972) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5805 0x16ad 0x240000400 1 5768 0x1688 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5806 0x16ae 0x240000400 1 5769 0x1689 0x280000400 rc1=0 and rc2=0 PASS 93 (19s) == sanityn test 94: signal vs CP callback race =========== 16:43:11 (1788208991) osc.lustre-OST0000-osc-ffff99d2c481a800.idle_timeout=0 osc.lustre-OST0000-osc-ffff99d2d0cf1800.idle_timeout=0 osc.lustre-OST0001-osc-ffff99d2c481a800.idle_timeout=0 osc.lustre-OST0001-osc-ffff99d2d0cf1800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.232794 s, 35.2 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.04656 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 290211 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.107811 s, 38.0 kB/s osc.lustre-OST0000-osc-ffff99d2c481a800.idle_timeout=debug osc.lustre-OST0000-osc-ffff99d2d0cf1800.idle_timeout=debug osc.lustre-OST0001-osc-ffff99d2c481a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff99d2d0cf1800.idle_timeout=debug PASS 94 (17s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 16:43:28 (1788209008) llite.lustre-ffff99d2c481a800.fast_read=0 llite.lustre-ffff99d2d0cf1800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.116909 s, 17.9 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff99d2c481a800.fast_read=1 llite.lustre-ffff99d2d0cf1800.fast_read=1 PASS 95a (20s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 16:43:48 (1788209028) llite.lustre-ffff99d2c481a800.fast_read=0 llite.lustre-ffff99d2d0cf1800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0106166 s, 1.2 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff99d2c481a800.fast_read=1 llite.lustre-ffff99d2d0cf1800.fast_read=1 PASS 95b (13s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 16:44:01 (1788209041) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 16:44:05 (1788209045) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00543486 s, 754 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (6s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 16:44:11 (1788209051) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.127345 s, 16.5 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (7s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 16:44:18 (1788209058) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.243997 s, 8.6 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (7s) == sanityn test 100e: DoM: read on open and file size ==== 16:44:25 (1788209065) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00863326 s, 1.4 MB/s PASS 100e (6s) == sanityn test 101a: Discard DoM data on unlink ========= 16:44:32 (1788209072) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0134131 s, 305 kB/s PASS 101a (6s) == sanityn test 101b: Discard DoM data on rename ========= 16:44:38 (1788209078) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0308888 s, 133 kB/s PASS 101b (6s) == sanityn test 101c: Discard DoM data on close-unlink === 16:44:44 (1788209084) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0100477 s, 408 kB/s PASS 101c (8s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 16:44:52 (1788209092) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 66 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf66:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (9s) == sanityn test 103: Test size correctness with lockahead ========================================================== 16:45:01 (1788209101) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788209104 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788209107 Finishing test test23 at 1788209108 PASS 103 (15s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 16:45:16 (1788209116) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 16:45:20 (1788209120) fail_loc=0x416 PASS 105 (24s) == sanityn test 106a: Verify the btime via statx() ======= 16:45:44 (1788209144) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 16:45:48 (1788209148) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0287087 s, 35.7 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938221 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-31 16:45:50.000000000 -0400 Modify: 2026-08-31 16:45:50.000000000 -0400 Change: 2026-08-31 16:45:50.000000000 -0400 Birth: 2026-08-31 16:45:49.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938221 -rw-r--r-- 1 0 0 1788209149 1788209150 1788209150 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938221 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-31 16:45:50.000000000 -0400 Modify: 2026-08-31 16:45:50.000000000 -0400 Change: 2026-08-31 16:45:50.000000000 -0400 Birth: 2026-08-31 16:45:49.000000000 -0400 1788209150 1024 1 PASS 106b (9s) == sanityn test 106c: Verify statx attributes mask ======= 16:45:57 (1788209157) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 16:46:05 (1788209165) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.414083 s, 25.3 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5815 0x16b7 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 First grouplock blocks second one PASS 107a (11s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 16:46:16 (1788209176) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.496968 s, 21.1 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5776 0x1690 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (14s) == sanityn test 108a: lseek: parallel updates ============ 16:46:30 (1788209190) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00959442 s, 1.0 MB/s llite.lustre-ffff99d2c481a800.hybrid_io=0 llite.lustre-ffff99d2d0cf1800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.6848 s, 1.8 MB/s PASS 108a (13s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 16:46:43 (1788209203) 192.168.204.158@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg458-client.virtnet /mnt/lustre (opts:) 192.168.204.158@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg458-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre2 Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre 192.168.204.158@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.158@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg458-client.virtnet /mnt/lustre (opts:) Stopping client oleg458-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg458-server' oleg458-server: oleg458-server.virtnet: executing load_modules_local oleg458-server: kmemleak disabled oleg458-server: Loading modules from /home/green/git/lustre-release/lustre oleg458-server: detected 4 online CPUs by sysfs oleg458-server: MODOPTS_LIBCFS= oleg458-server: Force libcfs to create 2 CPU partitions oleg458-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre2 Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre 192.168.204.158@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.158@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg458-client.virtnet /mnt/lustre (opts:) Stopping client oleg458-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg458-server' oleg458-server: oleg458-server.virtnet: executing load_modules_local oleg458-server: kmemleak disabled oleg458-server: Loading modules from /home/green/git/lustre-release/lustre oleg458-server: detected 4 online CPUs by sysfs oleg458-server: MODOPTS_LIBCFS= oleg458-server: Force libcfs to create 2 CPU partitions oleg458-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre2 PASS 109 (71s) == sanityn test 110: do not grant another lock on resend ========================================================== 16:47:54 (1788209274) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-31 16:47:56.000000000 -0400 Modify: 2026-08-31 16:47:56.000000000 -0400 Change: 2026-08-31 16:47:56.000000000 -0400 Birth: 2026-08-31 16:47:56.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg458-client.virtnet oleg458-client.virtnet PASS 110 (83s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 16:49:17 (1788209357) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (3s) == sanityn test 112: update max-inherit in default LMV === 16:49:21 (1788209361) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (3s) == sanityn test 113: check servers of specified fs ======= 16:49:24 (1788209364) lustre-OST0000-osc-ffff99d2d1dcc800 active. lustre-OST0001-osc-ffff99d2d1dcc800 active. lustre-OST0000-osc-ffff99d2d1dcc800 active. lustre-OST0001-osc-ffff99d2d1dcc800 active. MGC192.168.204.158@tcp active. PASS 113 (8s) == sanityn test 114: implicit default LMV inherit ======== 16:49:32 (1788209372) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (4s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 16:49:36 (1788209376) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (4s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 16:49:40 (1788209380) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (4s) == sanityn test 121: trunc append race =================== 16:49:44 (1788209384) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0276746 s, 740 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.121828 s, 168 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (10s) == sanityn test 122: directory size is consistent across mounts ========================================================== 16:49:54 (1788209394) PASS 122 (11s) == sanityn test 200: service remains healthy while able to process request ========================================================== 16:50:05 (1788209405) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1176 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1723 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2645 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2309 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2318 s, 0.2 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2583 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2967 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1743 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2415 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.286 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2745 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2857 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3108 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.346 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3634 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3835 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.285 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3285 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg458-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff99d2c5462000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99d2c5462000.ost_server_uuid in IDLE state after 0 sec PASS 200 (85s) cleanup: ====================================================== == sanityn test complete, duration 10632 sec ============= 16:51:32 (1788209492) === sanityn: start cleanup 16:51:34 (1788209494) === Stopping clients: oleg458-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg458-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 16:55:49 (1788209749) ===