-----============= acceptance-small: sanityn ============----- Fri Sep 4 17:59:52 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg624-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 18:00:11 (1788559211) === Starting client oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre2 Mount client oleg624-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre2 Started clients oleg624-client.virtnet: 192.168.206.124@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg624-client.virtnet: executing check_config_client /mnt/lustre oleg624-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg624-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff99a64362d800.idle_timeout=debug osc.lustre-OST0000-osc-ffff99a6529e6800.idle_timeout=debug osc.lustre-OST0001-osc-ffff99a64362d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff99a6529e6800.idle_timeout=debug disable quota as required oleg624-server: oleg624-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 18:00:41 (1788559241) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307625 s, 34.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7641] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 18:00:44 (1788559244) PASS 0 (10s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 18:00:54 (1788559254) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (12s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 18:01:06 (1788559266) -rw-r--r-- 1 root root 0 Sep 4 18:01 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (9s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 18:01:15 (1788559275) -rw-r--r-- 1 root root 0 Sep 4 18:01 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (8s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 18:01:23 (1788559283) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 18:01:31 (1788559291) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (8s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 18:01:38 (1788559298) total 1 -rwxrwxrwx 1 root root 0 Sep 4 18:01 f2a -rwxrwxrwx 1 root root 0 Sep 4 18:01 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 4 18:01 f2a -rwxrwxrwx 1 root root 0 Sep 4 18:01 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0112317 s, 45.6 kB/s PASS 2e (6s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 18:01:44 (1788559304) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 18:01:47 (1788559307) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.204965 s, 10.2 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (5s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 18:01:52 (1788559312) PASS 3 (5s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 18:01:57 (1788559317) PASS 4 (7s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 18:02:04 (1788559324) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (8s) == sanityn test 6: remove of open file on other node ============================================================================ 18:02:12 (1788559332) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (6s) == sanityn test 7: remove of open directory on other node ======================================================================= 18:02:18 (1788559338) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (7s) == sanityn test 8: remove of open special file on other node ==================================================================== 18:02:25 (1788559345) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (6s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 18:02:31 (1788559351) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 18:02:39 (1788559359) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117867 s, 8.9 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.105851 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (6s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 18:02:45 (1788559365) 1+0 records in 1+0 records out 1 byte copied, 0.0144458 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0251518 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0187235 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0200352 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.021145 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0224741 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00871912 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0133732 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0137709 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0101974 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0168444 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0137425 s, 0.1 kB/s PASS 10a (7s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 18:02:53 (1788559373) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0145503 s, 211 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0384397 s, 107 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000286858 s, 10.7 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 18:02:59 (1788559379) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7641 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 18:03:06 (1788559386) warning: '-runtime' deprecated, use '-t runtime' instead - link 320 (time 1788559398.46 total 10.05 last 31.84) - link 639 (time 1788559408.47 total 20.06 last 31.89) - link 943 (time 1788559418.50 total 30.09 last 30.29) - link 1261 (time 1788559428.52 total 40.10 last 31.76) - link 1622 (time 1788559438.53 total 50.12 last 36.04) - link 1998 (time 1788559448.55 total 60.14 last 37.54) - link 2351 (time 1788559458.55 total 70.14 last 35.30) - link 2716 (time 1788559468.56 total 80.15 last 36.45) - link 3085 (time 1788559478.57 total 90.16 last 36.86) total: 3466 link in 100.02 seconds: 34.65 ops/second using seed 1584695102 running for 100 seconds - stat 4077 (time 1788559400 ; total 11 ; last 11) - stat 9784 (time 1788559411 ; total 22 ; last 11) - stat 10000 (time 1788559411 ; total 22 ; last 0) - stat 14521 (time 1788559422 ; total 33 ; last 11) - stat 20000 (time 1788559431 ; total 42 ; last 9) - stat 25503 (time 1788559442 ; total 53 ; last 11) - stat 28449 (time 1788559453 ; total 64 ; last 11) - stat 30000 (time 1788559459 ; total 70 ; last 6) - stat 33049 (time 1788559470 ; total 81 ; last 11) - stat 36104 (time 1788559481 ; total 92 ; last 11) total: 38489 stats in 100 seconds: 384.890015 stats/second - unlinked 0 (time 1788559439 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3466) error: No such file or directory total: 3465 unlinks in 82 seconds: 42.256096 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21450) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21452) - No such process PASS 12 (140s) == sanityn test 13: test directory page revocation ======= 18:05:26 (1788559526) f13 f13 PASS 13 (7s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 18:05:33 (1788559533) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7641 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22636 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (5s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 18:05:38 (1788559538) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23215 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (6s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 18:05:44 (1788559544) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23793 Terminated $DIR1/$tdir/sleep 60 PASS 14b (6s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 18:05:50 (1788559550) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24376 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 18:05:57 (1788559557) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24959 Terminated $DIR1/$tdir/sleep 60 PASS 14d (11s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 18:06:09 (1788559569) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (46s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 18:06:55 (1788559615) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.570486 s, 14.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.44252 s, 19.0 MB/s Chance of close/open is 1 in 50 Seed set to 9619 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero length punch hole truncating to largest ever: 0x43b572 000006[0] 1788559620.090100 trunc from 00000000 to 0x43b571 (0x43b572 bytes) 000008[1] 1788559620.130377 trunc from 0x43b572 to 0x2f715f (0x144412 bytes) 000009[1] 1788559620.174387 read 0x2ce7bc thru 0x2dce9d (0xe6e2 bytes) 000010[1] 1788559620.188296 read 0x1d746a thru 0x1e365f (0xc1f6 bytes) 000011[1] 1788559620.274874 mapwrite 0x03261b thru 0x034dda (0x27c0 bytes) 000012[0] 1788559620.337675 read 0x14473b thru 0x14d3ac (0x8c72 bytes) 000015[1] 1788559620.374008 punch from 0x0a534a to 0x0b419d (0xee54 bytes) 000017[0] 1788559620.416019 mapread 0x1466c1 thru 0x14e3db (0x7d1b bytes) truncating to largest ever: 0x75fedb 000020[1] 1788559620.478201 trunc from 0x2f7160 to 0x75feda (0x468d7b bytes) 000022[0] 1788559620.525405 mapwrite 0x835ebc thru 0x83ef4c (0x9091 bytes) 000024[0] 1788559620.699640 write 0x1def08 thru 0x1e4040 (0x5139 bytes) 000025[1] 1788559620.739460 punch from 0x3e9a40 to 0x3ef2b7 (0x5878 bytes) 000028[0] 1788559620.828890 trunc from 0x83ef4d to 0x3bb6eb (0x483861 bytes) 000030[1] 1788559620.890653 trunc from 0x3bb6ec to 0x224983 (0x196d68 bytes) 000033[1] 1788559620.997053 mapread 0x13b860 thru 0x147b01 (0xc2a2 bytes) 000035[1] 1788559621.067519 read 0x00f2f2 thru 0x017a1d (0x872c bytes) 000038[0] 1788559621.106957 mapwrite 0x3fbbaf thru 0x402cef (0x7141 bytes) 000040[0] 1788559621.365473 read 0x05477d thru 0x05f547 (0xadcb bytes) 000041[1] 1788559621.400805 punch from 0x269089 to 0x26e0a2 (0x501a bytes) 000042[1] 1788559621.439441 read 0x0d61be thru 0x0e0dc7 (0xac0a bytes) 000043[1] 1788559621.461484 trunc from 0x402cf0 to 0x259ec3 (0x1a8e2c bytes) 000045[1] 1788559621.529043 mapwrite 0x4e2535 thru 0x4e63ec (0x3eb8 bytes) 000046[1] 1788559621.675821 mapwrite 0x7d70db thru 0x7dc97b (0x58a1 bytes) 000047[0] 1788559621.787700 trunc from 0x7dc97c to 0x41361c (0x3c935f bytes) 000049[1] 1788559621.867999 write 0x188b00 thru 0x18f5ac (0x6aad bytes) 000050[1] 1788559621.926368 read 0x010b45 thru 0x015a0e (0x4eca bytes) 000051[1] 1788559621.946426 mapread 0x14769f thru 0x151275 (0x9bd7 bytes) 000055[0] 1788559622.019542 mapread 0x0fffd8 thru 0x1093ae (0x93d7 bytes) 000057[0] 1788559622.060660 mapread 0x3a83a2 thru 0x3ac8c0 (0x451f bytes) 000058[0] 1788559622.075923 trunc from 0x41361d to 0x5259cb (0x1123af bytes) 000059[0] 1788559622.114055 write 0x9a0da4 thru 0x9a584b (0x4aa8 bytes) 000060[0] 1788559622.121314 mapread 0x85c1b8 thru 0x85d1b4 (0xffd bytes) truncating to largest ever: 0x9ec7dd 000061[0] 1788559622.148348 trunc from 0x9a584c to 0x9ec7dc (0x46f91 bytes) 000062[0] 1788559622.195930 read 0x9cf26c thru 0x9d54d7 (0x626c bytes) 000063[0] 1788559622.231639 write 0x9fb0a4 thru 0x9fffff (0x4f5c bytes) 000064[1] 1788559622.252008 punch from 0x5dca90 to 0x5dff38 (0x34a9 bytes) 000065[1] 1788559622.284010 punch from 0x8f2126 to 0x9008d6 (0xe7b1 bytes) 000066[1] 1788559622.332615 punch from 0x8d2cbc to 0x8dc915 (0x9c5a bytes) 000067[1] 1788559622.337842 mapwrite 0x730cd0 thru 0x730dda (0x10b bytes) 000069[1] 1788559622.425264 write 0x38ccc5 thru 0x39b127 (0xe463 bytes) 000070[1] 1788559622.453726 mapread 0x5b9aad thru 0x5c0ecc (0x7420 bytes) 000072[1] 1788559622.495878 mapwrite 0x389c96 thru 0x39073c (0x6aa7 bytes) 000076[1] 1788559622.564647 mapread 0x265cc8 thru 0x27096b (0xaca4 bytes) 000077[0] 1788559622.596612 trunc from 0xa00000 to 0x6ba101 (0x345efe bytes) 000080[0] 1788559622.670959 read 0x5ed29c thru 0x5f0172 (0x2ed7 bytes) 000082[1] 1788559622.708702 read 0x2db7a1 thru 0x2e9b87 (0xe3e7 bytes) 000085[0] 1788559622.761470 mapread 0x5a71d5 thru 0x5b3799 (0xc5c5 bytes) 000087[1] 1788559622.795841 write 0x3f200b thru 0x3fabfb (0x8bf1 bytes) 000091[0] 1788559622.874351 mapread 0x040784 thru 0x048bad (0x842a bytes) 000092[0] 1788559622.935617 mapwrite 0x588138 thru 0x5937ae (0xb677 bytes) 000095[0] 1788559623.037375 mapwrite 0x057fdd thru 0x065c8c (0xdcb0 bytes) 000096[0] 1788559623.126478 mapread 0x35d5bc thru 0x368988 (0xb3cd bytes) 000097[0] 1788559623.165198 trunc from 0x6ba102 to 0x635c9c (0x84465 bytes) 000098[0] 1788559623.207602 punch from 0x431a00 to 0x43c491 (0xaa92 bytes) 000099[1] 1788559623.229222 mapwrite 0x285afb thru 0x286c41 (0x1147 bytes) 000103[0] 1788559623.394870 punch from 0x009a46 to 0x0118f0 (0x7eab bytes) 000104[0] 1788559623.406475 mapwrite 0x9e00ba thru 0x9e9067 (0x8fae bytes) 000105[0] 1788559623.503134 trunc from 0x9e9068 to 0x00d150 (0x9dbf17 bytes) 000107[0] 1788559623.528710 mapread 0x008972 thru 0x00a174 (0x1803 bytes) 000109[0] 1788559623.615994 write 0x0ba559 thru 0x0c4a06 (0xa4ae bytes) 000110[1] 1788559623.618014 mapwrite 0x2903df thru 0x29bfb0 (0xbbd2 bytes) 000112[1] 1788559623.793166 punch from 0x189758 to 0x1982b7 (0xeb60 bytes) 000113[0] 1788559623.824801 read 0x17655d thru 0x179f4c (0x39f0 bytes) 000115[0] 1788559623.926679 punch from 0x05c7e1 to 0x060450 (0x3c70 bytes) 000116[1] 1788559623.962058 write 0x3800cb thru 0x38756f (0x74a5 bytes) 000121[1] 1788559624.011281 mapread 0x2d5946 thru 0x2db976 (0x6031 bytes) 000122[0] 1788559624.018772 trunc from 0x387570 to 0x2585da (0x12ef95 bytes) 000123[1] 1788559624.079154 read 0x026592 thru 0x030365 (0x9dd4 bytes) 000124[0] 1788559624.160794 trunc from 0x2585db to 0x5ba1c5 (0x361beb bytes) 000125[1] 1788559624.209818 mapread 0x28515c thru 0x290baa (0xba4f bytes) 000127[1] 1788559624.279772 trunc from 0x5ba1c6 to 0x1027f9 (0x4b79cc bytes) 000128[0] 1788559624.377379 trunc from 0x1027fa to 0x0a34fa (0x5f2ff bytes) 000129[1] 1788559624.441080 mapwrite 0x5f602f thru 0x5f8aae (0x2a80 bytes) 000131[1] 1788559624.619812 punch from 0x2ea0e6 to 0x2f900f (0xef2a bytes) 000132[1] 1788559624.645828 read 0x21b996 thru 0x2257e2 (0x9e4d bytes) 000133[1] 1788559624.654952 write 0x050628 thru 0x05d019 (0xc9f2 bytes) 000137[1] 1788559624.755610 trunc from 0x5f8aaf to 0x1e9bb3 (0x40eefb bytes) 000138[1] 1788559624.786537 write 0x9f79c9 thru 0x9ff0a1 (0x76d9 bytes) 000139[1] 1788559624.796517 read 0x6a2779 thru 0x6b1d87 (0xf60f bytes) 000141[1] 1788559624.835528 punch from 0x973f78 to 0x978ec3 (0x4f4c bytes) 000144[1] 1788559624.923865 read 0x26a890 thru 0x26d492 (0x2c03 bytes) 000145[0] 1788559624.946148 punch from 0x3cf4d2 to 0x3dbc6b (0xc79a bytes) 000147[1] 1788559625.012821 mapread 0x1fdc94 thru 0x2024ac (0x4819 bytes) 000151[1] 1788559625.066764 read 0x5ddfe0 thru 0x5edac6 (0xfae7 bytes) 000152[1] 1788559625.074987 mapread 0x09b9d8 thru 0x0aac23 (0xf24c bytes) 000153[1] 1788559625.143997 mapwrite 0x5f2400 thru 0x600930 (0xe531 bytes) 000154[0] 1788559625.369063 punch from 0x4fac47 to 0x4ff58e (0x4948 bytes) 000155[1] 1788559625.398032 punch from 0x6196d0 to 0x6259c6 (0xc2f7 bytes) 000158[0] 1788559625.501648 read 0x3ef08b thru 0x3f6b6e (0x7ae4 bytes) 000159[1] 1788559625.531332 punch from 0x3f4dc8 to 0x3ff6c6 (0xa8ff bytes) 000160[0] 1788559625.565834 write 0x24049c thru 0x245017 (0x4b7c bytes) 000161[0] 1788559625.596982 write 0x832fc4 thru 0x833f5d (0xf9a bytes) 000162[0] 1788559625.616646 read 0x64c815 thru 0x653957 (0x7143 bytes) 000164[0] 1788559625.689803 write 0x6992e1 thru 0x69e1a8 (0x4ec8 bytes) 000165[1] 1788559625.758245 punch from 0x7e8a3d to 0x7f403e (0xb602 bytes) 000166[0] 1788559625.806704 read 0x67f3b7 thru 0x68d740 (0xe38a bytes) 000167[1] 1788559625.854557 read 0x128a8f thru 0x12d2de (0x4850 bytes) 000168[1] 1788559625.876362 read 0x3b405a thru 0x3b793c (0x38e3 bytes) 000169[0] 1788559625.882334 write 0x2949d8 thru 0x29b788 (0x6db1 bytes) 000170[1] 1788559625.914595 mapwrite 0x5c4f59 thru 0x5d4664 (0xf70c bytes) 000171[0] 1788559626.041098 read 0x08c48c thru 0x0937f8 (0x736d bytes) 000172[0] 1788559626.061647 write 0x079864 thru 0x07e5f2 (0x4d8f bytes) 000174[0] 1788559626.119494 read 0x4cd54b thru 0x4cea4f (0x1505 bytes) 000175[1] 1788559626.137071 mapread 0x7c4ab2 thru 0x7ca657 (0x5ba6 bytes) 000177[0] 1788559626.328350 mapwrite 0x6bce44 thru 0x6be364 (0x1521 bytes) 000178[1] 1788559626.459584 mapread 0x871b3d thru 0x87fc55 (0xe119 bytes) 000179[1] 1788559626.513855 mapwrite 0x841ab7 thru 0x8500b3 (0xe5fd bytes) 000182[0] 1788559626.678345 mapwrite 0x4ab390 thru 0x4baae0 (0xf751 bytes) 000183[1] 1788559626.792459 trunc from 0x9ff0a2 to 0x74f6e3 (0x2af9be bytes) 000185[0] 1788559626.851869 write 0x57e274 thru 0x5838cf (0x565c bytes) 000186[0] 1788559626.897555 mapread 0x36a919 thru 0x372ca9 (0x8391 bytes) 000187[1] 1788559626.952676 write 0x42b1ae thru 0x4302a4 (0x50f7 bytes) 000188[1] 1788559627.025312 trunc from 0x74f6e4 to 0x75e2c5 (0xebe2 bytes) 000189[1] 1788559627.059624 mapread 0x63e58a thru 0x649a59 (0xb4d0 bytes) 000190[0] 1788559627.189726 write 0x11e249 thru 0x11ed8e (0xb46 bytes) 000191[1] 1788559627.239671 write 0x72bb31 thru 0x72c172 (0x642 bytes) 000192[0] 1788559627.261578 punch from 0x691411 to 0x6a0003 (0xebf3 bytes) 000195[1] 1788559627.333602 mapwrite 0x56c515 thru 0x56e771 (0x225d bytes) 000196[0] 1788559627.417100 mapread 0x0a469f thru 0x0aa63a (0x5f9c bytes) 000197[0] 1788559627.453657 mapwrite 0x8f68b4 thru 0x90116b (0xa8b8 bytes) 000198[1] 1788559627.595583 trunc from 0x90116c to 0x901dae (0xc43 bytes) 000199[1] 1788559627.647045 write 0x9e8a72 thru 0x9f710f (0xe69e bytes) 000203[0] 1788559627.737388 trunc from 0x9f7110 to 0x6a072c (0x3569e3 bytes) 000204[0] 1788559627.788865 mapwrite 0x362d3e thru 0x370a45 (0xdd08 bytes) 000206[1] 1788559627.867014 mapread 0x39791e thru 0x39e3b0 (0x6a93 bytes) 000209[1] 1788559627.965575 mapread 0x46c052 thru 0x47421c (0x81cb bytes) 000212[1] 1788559628.076520 punch from 0x3dedf1 to 0x3e700f (0x821f bytes) 000214[1] 1788559628.104720 trunc from 0x6a072d to 0x1c913e (0x4d75ee bytes) 000216[0] 1788559628.141460 read 0x05c557 thru 0x06c1e7 (0xfc91 bytes) 000223[0] 1788559628.194528 punch from 0x179102 to 0x184cf0 (0xbbef bytes) 000226[0] 1788559628.306783 mapread 0x07a322 thru 0x08949b (0xf17a bytes) 000227[1] 1788559628.382941 mapread 0x0d8574 thru 0x0e3f55 (0xb9e2 bytes) 000228[0] 1788559628.537797 punch from 0x07691f to 0x077ebf (0x15a1 bytes) 000231[1] 1788559628.644226 mapwrite 0x54f7ec thru 0x55073f (0xf54 bytes) 000232[0] 1788559628.736706 trunc from 0x550740 to 0x705c48 (0x1b5509 bytes) 000233[1] 1788559628.790542 trunc from 0x705c49 to 0x1d011b (0x535b2d bytes) 000234[1] 1788559628.835178 read 0x10cceb thru 0x10ded3 (0x11e9 bytes) 000235[1] 1788559628.927439 read 0x064a0a thru 0x06dcb5 (0x92ac bytes) 000236[0] 1788559628.943240 mapwrite 0x9b9354 thru 0x9c01bb (0x6e68 bytes) 000237[1] 1788559629.052964 mapwrite 0x7bd7a1 thru 0x7c0a83 (0x32e3 bytes) 000239[1] 1788559629.185789 read 0x79a08b thru 0x79eff5 (0x4f6b bytes) 000241[1] 1788559629.232415 punch from 0x3dda4d to 0x3ed77d (0xfd31 bytes) 000242[0] 1788559629.276613 mapwrite 0x034751 thru 0x03776c (0x301c bytes) 000243[0] 1788559629.371140 mapread 0x3a6d39 thru 0x3ae276 (0x753e bytes) 000246[1] 1788559629.439639 mapread 0x1fe59c thru 0x201479 (0x2ede bytes) 000248[0] 1788559629.493444 mapwrite 0x7e68f2 thru 0x7e92be (0x29cd bytes) 000249[0] 1788559629.568773 write 0x8e8f6b thru 0x8f24fd (0x9593 bytes) 000250[1] 1788559629.613582 write 0x2c5f67 thru 0x2ca231 (0x42cb bytes) 000251[1] 1788559629.664591 mapread 0x720eb3 thru 0x7212d9 (0x427 bytes) 000256[1] 1788559629.709105 read 0x3bd84c thru 0x3cd457 (0xfc0c bytes) 000257[1] 1788559629.716818 mapread 0x5f29cd thru 0x5f2c97 (0x2cb bytes) 000258[1] 1788559629.722192 write 0x5a1b85 thru 0x5b0f4e (0xf3ca bytes) 000259[0] 1788559629.757791 read 0x80cbac thru 0x818de6 (0xc23b bytes) 000261[0] 1788559629.815405 write 0x364d9e thru 0x36d04a (0x82ad bytes) 000263[1] 1788559629.873724 punch from 0x0e072e to 0x0e1dc1 (0x1694 bytes) 000265[0] 1788559629.943382 punch from 0x2d0cc9 to 0x2d4a8b (0x3dc3 bytes) 000266[0] 1788559630.000676 mapwrite 0x20c806 thru 0x20d4d3 (0xcce bytes) 000267[0] 1788559630.080445 mapwrite 0x5851f5 thru 0x5854b4 (0x2c0 bytes) 000268[1] 1788559630.167988 mapread 0x44bcea thru 0x450a49 (0x4d60 bytes) 000269[0] 1788559630.225835 read 0x042889 thru 0x0498bc (0x7034 bytes) 000270[1] 1788559630.247633 mapwrite 0x56ae2d thru 0x571a30 (0x6c04 bytes) 000271[0] 1788559630.399330 trunc from 0x9c01bc to 0x6cb89b (0x2f4920 bytes) 000274[1] 1788559630.437124 write 0x67ee06 thru 0x68d18d (0xe388 bytes) 000275[0] 1788559630.486699 read 0x3697e7 thru 0x370192 (0x69ac bytes) 000276[1] 1788559630.507058 trunc from 0x6cb89c to 0x380281 (0x34b61a bytes) 000278[0] 1788559630.566457 punch from 0x2c792b to 0x2ca62b (0x2d01 bytes) 000280[1] 1788559630.606136 mapread 0x2adb25 thru 0x2b85e4 (0xaac0 bytes) 000281[0] 1788559630.668643 write 0x109067 thru 0x10a77d (0x1717 bytes) 000287[1] 1788559630.767435 punch from 0x22ceac to 0x2332d5 (0x642a bytes) 000289[0] 1788559630.829609 read 0x0607b1 thru 0x06536a (0x4bba bytes) 000291[0] 1788559630.905315 mapread 0x226c7d thru 0x22ca98 (0x5e1c bytes) 000292[1] 1788559630.916158 write 0x821431 thru 0x82b759 (0xa329 bytes) 000295[0] 1788559630.979850 mapwrite 0x9b78a1 thru 0x9b7f38 (0x698 bytes) 000299[0] 1788559631.192657 mapread 0x511171 thru 0x512812 (0x16a2 bytes) 000300[1] 1788559631.200447 punch from 0x67a366 to 0x67cdd0 (0x2a6b bytes) 000301[0] 1788559631.252192 punch from 0x52915a to 0x535ad6 (0xc97d bytes) 000302[0] 1788559631.276060 trunc from 0x9b7f39 to 0x5c43a0 (0x3f3b98 bytes) 000307[1] 1788559631.384244 mapwrite 0x05569d thru 0x057079 (0x19dd bytes) 000308[1] 1788559631.530758 mapwrite 0x94ea02 thru 0x95263b (0x3c3a bytes) 000309[0] 1788559631.678708 trunc from 0x95263c to 0x540e38 (0x411803 bytes) 000310[0] 1788559631.739606 mapwrite 0x5ecccb thru 0x5f2894 (0x5bca bytes) 000311[1] 1788559631.863668 mapread 0x594697 thru 0x5970a7 (0x2a11 bytes) 000312[0] 1788559631.903970 write 0x753d8d thru 0x757026 (0x329a bytes) 000313[0] 1788559631.917860 punch from 0x507047 to 0x508b67 (0x1b21 bytes) 000314[0] 1788559631.924595 write 0x8273fb thru 0x82b04f (0x3c55 bytes) 000318[1] 1788559631.967729 mapwrite 0x8856c2 thru 0x8885e7 (0x2f26 bytes) 000319[1] 1788559632.123421 mapread 0x66aa22 thru 0x66bb1b (0x10fa bytes) 000320[1] 1788559632.141496 trunc from 0x8885e8 to 0x56917a (0x31f46d bytes) 000321[0] 1788559632.170876 trunc from 0x56917b to 0x7b5553 (0x24c3d9 bytes) 000322[0] 1788559632.206660 write 0x7a7700 thru 0x7adff3 (0x68f4 bytes) 000323[0] 1788559632.241792 read 0x2a1b0b thru 0x2a6e80 (0x5376 bytes) 000324[0] 1788559632.249877 read 0x027bda thru 0x0338da (0xbd01 bytes) 000325[0] 1788559632.278091 trunc from 0x7b5554 to 0x1e85b0 (0x5ccfa3 bytes) 000328[0] 1788559632.358556 write 0x94a3e2 thru 0x95017e (0x5d9d bytes) 000330[1] 1788559632.382664 mapwrite 0x9e0968 thru 0x9e1043 (0x6dc bytes) 000331[1] 1788559632.481015 trunc from 0x9e1044 to 0x8acf65 (0x1340de bytes) 000332[0] 1788559632.496284 write 0x0f203c thru 0x0fa920 (0x88e5 bytes) 000333[0] 1788559632.573979 write 0x4fbe0c thru 0x5013e2 (0x55d7 bytes) 000334[0] 1788559632.605622 mapwrite 0x8f27ad thru 0x90155c (0xedb0 bytes) 000335[0] 1788559632.766383 mapwrite 0x43fb4e thru 0x445f85 (0x6438 bytes) 000339[1] 1788559633.002169 read 0x74ac33 thru 0x74afbd (0x38b bytes) 000341[0] 1788559633.071274 read 0x5c122d thru 0x5c5058 (0x3e2c bytes) 000343[1] 1788559633.117275 write 0x8c16da thru 0x8c7190 (0x5ab7 bytes) 000344[1] 1788559633.159612 read 0x22a082 thru 0x237299 (0xd218 bytes) 000345[1] 1788559633.174231 mapread 0x0f0a18 thru 0x0f51a5 (0x478e bytes) 000347[0] 1788559633.262494 punch from 0x4019a6 to 0x41067b (0xecd6 bytes) 000348[0] 1788559633.296768 mapwrite 0x017aa9 thru 0x018400 (0x958 bytes) 000349[1] 1788559633.403012 read 0x6a9986 thru 0x6a9a5a (0xd5 bytes) 000351[1] 1788559633.439878 trunc from 0x90155d to 0x6aaab5 (0x256aa7 bytes) 000353[1] 1788559633.484011 read 0x399c08 thru 0x3a6624 (0xca1d bytes) 000354[0] 1788559633.521432 punch from 0x2462f2 to 0x2474dd (0x11ec bytes) 000355[0] 1788559633.564478 punch from 0x5f5a11 to 0x5fa2a7 (0x4897 bytes) 000356[0] 1788559633.580705 punch from 0x39a4d6 to 0x3a8b95 (0xe6c0 bytes) 000358[0] 1788559633.623848 read 0x2b2ea4 thru 0x2b539c (0x24f9 bytes) 000360[1] 1788559633.682075 mapread 0x4eb2e2 thru 0x4eddcc (0x2aeb bytes) 000361[1] 1788559633.726815 mapread 0x5e1791 thru 0x5e4afe (0x336e bytes) 000362[0] 1788559633.757991 write 0x5a5533 thru 0x5aaa4d (0x551b bytes) 000364[1] 1788559633.797544 punch from 0x4c329a to 0x4c4a99 (0x1800 bytes) 000366[0] 1788559633.853766 write 0x01105e thru 0x017416 (0x63b9 bytes) 000367[0] 1788559633.874086 read 0x392972 thru 0x39eff2 (0xc681 bytes) 000368[1] 1788559633.893319 write 0x071826 thru 0x08040c (0xebe7 bytes) 000370[0] 1788559633.935785 mapread 0x4fe861 thru 0x501cfa (0x349a bytes) 000371[0] 1788559633.982108 read 0x229347 thru 0x230f28 (0x7be2 bytes) 000374[0] 1788559634.015027 punch from 0x35df9c to 0x35e00d (0x72 bytes) 000378[1] 1788559634.050361 trunc from 0x6aaab6 to 0x22a162 (0x480953 bytes) 000380[1] 1788559634.116703 read 0x12b94b thru 0x134d94 (0x944a bytes) 000384[0] 1788559634.235695 mapread 0x0c69c4 thru 0x0cdda5 (0x73e2 bytes) 000385[0] 1788559634.326202 mapread 0x0ffdda thru 0x10878b (0x89b2 bytes) 000387[0] 1788559634.355970 write 0x9453fc thru 0x94bd4f (0x6954 bytes) 000388[0] 1788559634.394824 punch from 0x65315c to 0x6600ed (0xcf92 bytes) 000391[0] 1788559634.411886 write 0x088648 thru 0x09305f (0xaa18 bytes) 000392[0] 1788559634.451871 trunc from 0x94bd50 to 0x372ebe (0x5d8e91 bytes) 000396[1] 1788559634.521308 trunc from 0x372ebf to 0x38f91f (0x1ca61 bytes) 000397[0] 1788559634.574876 mapwrite 0x1a6086 thru 0x1b1c79 (0xbbf4 bytes) 000398[1] 1788559634.704700 read 0x11ca50 thru 0x12a9ee (0xdf9f bytes) 000400[1] 1788559634.789701 mapread 0x34c3df thru 0x352b65 (0x6787 bytes) 000403[1] 1788559634.822661 punch from 0x23cbf1 to 0x248a9e (0xbeae bytes) 000405[1] 1788559634.850105 mapwrite 0x22f56f thru 0x2349d0 (0x5462 bytes) 000408[0] 1788559635.067413 mapread 0x0a6d02 thru 0x0a7d54 (0x1053 bytes) 000410[0] 1788559635.119787 mapread 0x3104f8 thru 0x319ca0 (0x97a9 bytes) 000411[0] 1788559635.170050 punch from 0x22ea6c to 0x23a95e (0xbef3 bytes) 000412[0] 1788559635.212564 punch from 0x0bb172 to 0x0c6e07 (0xbc96 bytes) 000414[1] 1788559635.256722 punch from 0x2988fe to 0x2a3baa (0xb2ad bytes) 000415[1] 1788559635.291719 mapwrite 0x453d52 thru 0x4542da (0x589 bytes) 000416[1] 1788559635.395345 mapwrite 0x4bdc56 thru 0x4cdbc6 (0xff71 bytes) 000417[1] 1788559635.501740 mapread 0x031eb3 thru 0x03adde (0x8f2c bytes) 000421[1] 1788559635.593885 punch from 0x3f2e7d to 0x3f6272 (0x33f6 bytes) 000424[0] 1788559635.623524 write 0x0ffbba thru 0x100104 (0x54b bytes) 000425[1] 1788559635.661564 read 0x30973d thru 0x312746 (0x900a bytes) 000426[1] 1788559635.732822 trunc from 0x4cdbc7 to 0x0f0baf (0x3dd017 bytes) 000427[0] 1788559635.818437 mapwrite 0x8c7ac1 thru 0x8cc3c1 (0x4901 bytes) 000428[1] 1788559635.968690 mapread 0x6e1e00 thru 0x6e48ff (0x2b00 bytes) 000429[0] 1788559636.056244 trunc from 0x8cc3c2 to 0x3f8c28 (0x4d3799 bytes) 000430[0] 1788559636.103782 trunc from 0x3f8c29 to 0x3757a0 (0x83488 bytes) 000435[1] 1788559636.210637 read 0x314195 thru 0x3189c2 (0x482e bytes) 000436[0] 1788559636.348575 mapwrite 0x98d87b thru 0x99523f (0x79c5 bytes) 000440[0] 1788559636.614199 trunc from 0x995240 to 0x82731d (0x16df22 bytes) 000441[0] 1788559636.638584 mapread 0x7e31b0 thru 0x7e81e9 (0x503a bytes) 000442[1] 1788559636.702490 punch from 0x2b01ff to 0x2c0062 (0xfe64 bytes) 000444[1] 1788559636.762730 punch from 0x21af57 to 0x224fe8 (0xa092 bytes) 000446[1] 1788559636.800067 mapread 0x4db800 thru 0x4e16a3 (0x5ea4 bytes) 000449[1] 1788559636.912990 mapread 0x77184f thru 0x77ebd2 (0xd384 bytes) 000450[0] 1788559636.954130 write 0x3da6a1 thru 0x3e85ab (0xdf0b bytes) 000451[0] 1788559637.002785 read 0x6bb54b thru 0x6bf24c (0x3d02 bytes) 000452[0] 1788559637.055848 mapwrite 0x7a882a thru 0x7b57ac (0xcf83 bytes) 000456[1] 1788559637.214076 read 0x811411 thru 0x81e499 (0xd089 bytes) 000457[0] 1788559637.264786 punch from 0x2f5e7d to 0x2f8bf7 (0x2d7b bytes) 000458[0] 1788559637.277138 read 0x5e5704 thru 0x5f0fc3 (0xb8c0 bytes) 000459[0] 1788559637.315090 mapread 0x573bfc thru 0x57b0ee (0x74f3 bytes) 000460[1] 1788559637.378474 read 0x5c276c thru 0x5d1d71 (0xf606 bytes) 000464[0] 1788559637.441615 punch from 0x6a8251 to 0x6af707 (0x74b7 bytes) 000467[1] 1788559637.470819 write 0x850106 thru 0x851202 (0x10fd bytes) 000468[1] 1788559637.495674 punch from 0x4cc9f3 to 0x4d3daf (0x73bd bytes) 000471[1] 1788559637.548566 punch from 0x185a1c to 0x195295 (0xf87a bytes) 000475[0] 1788559637.591072 read 0x1f6c13 thru 0x1fd9e8 (0x6dd6 bytes) 000476[0] 1788559637.618647 mapwrite 0x6660e7 thru 0x66d346 (0x7260 bytes) 000477[1] 1788559637.729709 mapwrite 0x352939 thru 0x3625f9 (0xfcc1 bytes) 000478[0] 1788559637.859222 trunc from 0x851203 to 0x05ada3 (0x7f645f bytes) 000480[0] 1788559637.917248 trunc from 0x05ada4 to 0x76abc9 (0x70fe26 bytes) 000481[1] 1788559637.952890 write 0x997ec2 thru 0x9a1376 (0x94b5 bytes) 000484[0] 1788559638.013864 write 0x3c099c thru 0x3ca568 (0x9bcd bytes) 000485[1] 1788559638.067299 read 0x7a39de thru 0x7b3203 (0xf826 bytes) 000486[0] 1788559638.247087 punch from 0x1c3f58 to 0x1cf7cb (0xb874 bytes) 000487[0] 1788559638.259069 write 0x075a5c thru 0x079fb7 (0x455c bytes) 000489[0] 1788559638.312956 mapread 0x6dbead thru 0x6e78b9 (0xba0d bytes) 000490[1] 1788559638.345276 punch from 0x48f43d to 0x49a192 (0xad56 bytes) 000491[0] 1788559638.375936 read 0x4958f6 thru 0x49a1ec (0x48f7 bytes) 000492[0] 1788559638.414112 write 0x537328 thru 0x5411bb (0x9e94 bytes) 000494[1] 1788559638.488284 mapread 0x4fa716 thru 0x4fef46 (0x4831 bytes) 000495[0] 1788559638.554928 mapread 0x6f3472 thru 0x6fbafe (0x868d bytes) 000497[0] 1788559638.582724 mapread 0x5f6063 thru 0x603528 (0xd4c6 bytes) 000498[1] 1788559638.609221 write 0x6cf085 thru 0x6ddeb4 (0xee30 bytes) 000499[0] 1788559638.648374 write 0x0a7ead thru 0x0b3677 (0xb7cb bytes) 000500[1] 1788559638.670422 write 0x8eb2de thru 0x8f76e3 (0xc406 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 9638 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x5c7346 000005[0] 1788559639.693799 trunc from 00000000 to 0x5c7345 (0x5c7346 bytes) 000006[1] 1788559639.760873 write_OD 0x3d9000 thru 0x3d9fff (0x1000 bytes) 000007[0] 1788559639.815638 write 0x0ee000 thru 0x0f0f11 (0x2f12 bytes) 000008[0] 1788559639.883272 mapread 0x2a4000 thru 0x2af645 (0xb646 bytes) 000009[0] 1788559639.918757 mapwrite 0x2ec000 thru 0x2f98f0 (0xd8f1 bytes) truncating to largest ever: 0x6fd95d 000010[0] 1788559639.995015 trunc from 0x5c7346 to 0x6fd95c (0x136617 bytes) 000011[1] 1788559640.034795 punch from 0x6b423a to 0x6b8ef5 (0x4cbc bytes) 000012[0] 1788559640.073267 mapwrite 0x1dd000 thru 0x1ea4e0 (0xd4e1 bytes) 000013[0] 1788559640.237256 mapread 0x48c000 thru 0x491f81 (0x5f82 bytes) 000014[0] 1788559640.266083 mapwrite 0x4ae000 thru 0x4af586 (0x1587 bytes) 000016[1] 1788559640.400711 punch from 0x6b33f5 to 0x6b3b46 (0x752 bytes) 000017[1] 1788559640.437040 trunc from 0x6fd95d to 0x0b6440 (0x64751c bytes) 000018[1] 1788559640.477331 mapread 0x03a000 thru 0x047013 (0xd014 bytes) 000020[0] 1788559640.542337 trunc from 0x0b6441 to 0x3d8dc1 (0x322981 bytes) 000021[0] 1788559640.577079 mapwrite 0x77e000 thru 0x7825cd (0x45ce bytes) 000022[1] 1788559640.681356 read_OD 0x297000 thru 0x297fff (0x1000 bytes) 000023[0] 1788559640.729562 mapread 0x5b0000 thru 0x5b920c (0x920d bytes) 000025[1] 1788559640.778579 punch from 0x17a6d7 to 0x17bf71 (0x189b bytes) 000027[1] 1788559640.825375 write_OD 0x49c000 thru 0x49cfff (0x1000 bytes) 000028[0] 1788559640.878805 read 0x1eb000 thru 0x1fab8e (0xfb8f bytes) truncating to largest ever: 0x9980e7 000030[0] 1788559640.895058 trunc from 0x7825ce to 0x9980e6 (0x215b19 bytes) 000034[1] 1788559640.996930 mapwrite 0x640000 thru 0x64c390 (0xc391 bytes) 000035[1] 1788559641.103238 mapwrite 0x953000 thru 0x95bd4c (0x8d4d bytes) 000037[0] 1788559641.171769 trunc from 0x9980e7 to 0x7b3efa (0x1e41ec bytes) 000040[0] 1788559641.226622 write 0x1a8000 thru 0x1b4ce1 (0xcce2 bytes) 000041[1] 1788559641.242410 read_OD 0x1a2000 thru 0x1adfff (0xc000 bytes) 000046[0] 1788559641.279979 write 0x99d000 thru 0x9a18f6 (0x48f7 bytes) 000047[1] 1788559641.305580 read_OD 0x0e4000 thru 0x0f0fff (0xd000 bytes) 000048[0] 1788559641.331669 mapread 0x3a6000 thru 0x3a83f3 (0x23f4 bytes) 000049[1] 1788559641.351411 mapwrite 0x9b9000 thru 0x9bf3a1 (0x63a2 bytes) 000050[0] 1788559641.478527 trunc from 0x9bf3a2 to 0x01980e (0x9a5b93 bytes) 000052[1] 1788559641.589476 trunc from 0x01980f to 0x2d1970 (0x2b8162 bytes) 000053[0] 1788559641.657626 write 0x487000 thru 0x48f3ca (0x83cb bytes) 000054[0] 1788559641.690407 mapread 0x205000 thru 0x208da6 (0x3da7 bytes) 000055[0] 1788559641.759413 read 0x1d9000 thru 0x1e4d67 (0xbd68 bytes) 000057[0] 1788559641.841097 mapwrite 0x68b000 thru 0x6934b8 (0x84b9 bytes) 000059[1] 1788559642.073305 trunc from 0x6934b9 to 0x8cc58c (0x2390d4 bytes) 000060[0] 1788559642.114223 write 0x9a8000 thru 0x9aff67 (0x7f68 bytes) 000064[0] 1788559642.170735 mapread 0x00c000 thru 0x00e6dd (0x26de bytes) 000068[0] 1788559642.217860 mapwrite 0x4c6000 thru 0x4d06d1 (0xa6d2 bytes) truncating to largest ever: 0x9b7861 000072[1] 1788559642.335397 trunc from 0x9aff68 to 0x9b7860 (0x78f9 bytes) 000073[1] 1788559642.371703 write_OD 0x919000 thru 0x921fff (0x9000 bytes) 000074[0] 1788559642.461062 read 0x095000 thru 0x098982 (0x3983 bytes) 000075[1] 1788559642.484985 read_OD 0x80d000 thru 0x817fff (0xb000 bytes) 000077[0] 1788559642.503948 trunc from 0x9b7861 to 0x30671c (0x6b1144 bytes) 000080[0] 1788559642.599987 trunc from 0x30671d to 0x4f4bcd (0x1ee4b1 bytes) 000081[1] 1788559642.627205 write_OD 0x27b000 thru 0x289fff (0xf000 bytes) 000082[1] 1788559642.678071 trunc from 0x4f4bce to 0x295167 (0x25fa66 bytes) 000084[0] 1788559642.724191 trunc from 0x295168 to 0x6b437b (0x41f214 bytes) 000085[1] 1788559642.751691 mapwrite 0x76c000 thru 0x77b0b3 (0xf0b4 bytes) 000086[1] 1788559642.849879 write_OD 0x666000 thru 0x66efff (0x9000 bytes) 000089[0] 1788559642.900669 read 0x37d000 thru 0x382d13 (0x5d14 bytes) 000090[1] 1788559642.906835 punch from 0x5c150f to 0x5c66cf (0x51c1 bytes) 000092[0] 1788559642.950531 mapwrite 0x0e2000 thru 0x0ea4f5 (0x84f6 bytes) 000096[1] 1788559643.094438 mapread 0x455000 thru 0x463724 (0xe725 bytes) 000098[1] 1788559643.136871 mapwrite 0x7d3000 thru 0x7d9a72 (0x6a73 bytes) 000099[1] 1788559643.233683 mapwrite 0x768000 thru 0x7702d4 (0x82d5 bytes) 000102[0] 1788559643.360599 mapread 0x0b5000 thru 0x0b95fc (0x45fd bytes) 000104[1] 1788559643.393206 read_OD 0x6eb000 thru 0x6effff (0x5000 bytes) 000105[0] 1788559643.399404 read 0x0cb000 thru 0x0d8d72 (0xdd73 bytes) 000106[0] 1788559643.409642 trunc from 0x7d9a73 to 0x40a2b2 (0x3cf7c0 bytes) 000108[1] 1788559643.438874 punch from 0x2c6799 to 0x2c6d96 (0x5fe bytes) 000109[1] 1788559643.465794 mapwrite 0x9ae000 thru 0x9b3409 (0x540a bytes) 000110[0] 1788559643.539736 read 0x0a4000 thru 0x0a6382 (0x2383 bytes) 000113[1] 1788559643.557412 mapread 0x669000 thru 0x66c455 (0x3456 bytes) 000115[0] 1788559643.577550 read 0x4fd000 thru 0x4fddbd (0xdbe bytes) 000116[0] 1788559643.581237 punch from 0x5b98b8 to 0x5bd3e3 (0x3b2c bytes) 000119[1] 1788559643.632759 mapwrite 0x55f000 thru 0x55f1bb (0x1bc bytes) 000120[0] 1788559643.713636 mapwrite 0x76e000 thru 0x77bea0 (0xdea1 bytes) 000122[0] 1788559643.844423 punch from 0x6b2199 to 0x6c0fc8 (0xee30 bytes) 000123[0] 1788559643.859944 trunc from 0x9b340a to 0x5b5f25 (0x3fd4e4 bytes) 000124[0] 1788559643.890502 mapread 0x264000 thru 0x273641 (0xf642 bytes) 000128[1] 1788559643.941345 mapread 0x32a000 thru 0x336089 (0xc08a bytes) 000129[1] 1788559644.013403 trunc from 0x5b5f26 to 0x396f5e (0x21efc7 bytes) 000130[0] 1788559644.046089 mapwrite 0x36c000 thru 0x3708d1 (0x48d2 bytes) 000132[0] 1788559644.151661 mapread 0x0f2000 thru 0x0fb4e3 (0x94e4 bytes) 000134[0] 1788559644.170835 trunc from 0x396f5f to 0x4b969f (0x122741 bytes) 000135[0] 1788559644.195739 trunc from 0x4b96a0 to 0x721d02 (0x268663 bytes) 000139[1] 1788559644.243349 mapread 0x347000 thru 0x3533fe (0xc3ff bytes) 000140[1] 1788559644.279747 punch from 0x33a728 to 0x343085 (0x895e bytes) 000141[1] 1788559644.322131 mapread 0x506000 thru 0x509514 (0x3515 bytes) 000144[0] 1788559644.347834 trunc from 0x721d03 to 0x5e60ed (0x13bc15 bytes) 000147[1] 1788559644.410849 mapread 0x0ff000 thru 0x0ffc16 (0xc17 bytes) 000148[1] 1788559644.431364 trunc from 0x5e60ee to 0x5cad86 (0x1b367 bytes) 000149[1] 1788559644.473746 read_OD 0x15e000 thru 0x16bfff (0xe000 bytes) 000151[1] 1788559644.489758 trunc from 0x5cad87 to 0x1d0345 (0x3faa41 bytes) 000153[1] 1788559644.524966 write_OD 0x6f5000 thru 0x6fdfff (0x9000 bytes) 000155[0] 1788559644.677934 write 0x7b8000 thru 0x7be888 (0x6889 bytes) truncating to largest ever: 0x9cd363 000157[1] 1788559644.715208 trunc from 0x7be889 to 0x9cd362 (0x20eada bytes) 000159[1] 1788559644.824607 trunc from 0x9cd363 to 0x828282 (0x1a50e0 bytes) 000160[0] 1788559644.851397 read 0x01d000 thru 0x021f5b (0x4f5c bytes) 000163[0] 1788559644.882623 trunc from 0x828283 to 0x8cfba5 (0xa7923 bytes) 000164[0] 1788559644.939319 mapread 0x8c6000 thru 0x8cd676 (0x7677 bytes) 000165[0] 1788559644.990091 mapwrite 0x516000 thru 0x522586 (0xc587 bytes) 000168[1] 1788559645.079825 trunc from 0x8cfba6 to 0x35e6be (0x5714e7 bytes) 000171[1] 1788559645.138359 read_OD 0x22c000 thru 0x235fff (0xa000 bytes) 000172[1] 1788559645.153230 punch from 0x1ca4a4 to 0x1d57b6 (0xb313 bytes) 000173[1] 1788559645.178964 write_OD 0x576000 thru 0x57cfff (0x7000 bytes) 000176[1] 1788559645.239339 mapwrite 0x3f9000 thru 0x404147 (0xb148 bytes) 000177[0] 1788559645.404995 read 0x120000 thru 0x12c64c (0xc64d bytes) 000178[0] 1788559645.430808 mapread 0x344000 thru 0x348366 (0x4367 bytes) 000179[1] 1788559645.451421 mapread 0x3e0000 thru 0x3e70f7 (0x70f8 bytes) 000180[1] 1788559645.485160 read_OD 0x269000 thru 0x275fff (0xd000 bytes) 000181[0] 1788559645.492024 trunc from 0x57d000 to 0x5610b6 (0x1bf49 bytes) 000182[1] 1788559645.529665 mapwrite 0x15d000 thru 0x1617c8 (0x47c9 bytes) 000183[1] 1788559645.608541 mapwrite 0x285000 thru 0x285d19 (0xd1a bytes) 000184[0] 1788559645.649604 punch from 0x1d1545 to 0x1d9ecb (0x8987 bytes) 000185[0] 1788559645.713789 trunc from 0x5610b7 to 0x7f99d2 (0x29891c bytes) 000186[0] 1788559645.747087 trunc from 0x7f99d3 to 0x667e2a (0x191ba8 bytes) 000188[0] 1788559645.806475 read 0x11c000 thru 0x123483 (0x7484 bytes) 000189[1] 1788559645.832743 mapwrite 0x45e000 thru 0x468e13 (0xae14 bytes) 000190[1] 1788559645.938422 mapread 0x32a000 thru 0x335cea (0xbceb bytes) 000192[0] 1788559645.974716 mapwrite 0x6d1000 thru 0x6d1b1c (0xb1d bytes) 000193[1] 1788559646.057133 read_OD 0x3a4000 thru 0x3aafff (0x7000 bytes) 000196[0] 1788559646.094563 punch from 0x3b6992 to 0x3bfbae (0x921d bytes) 000199[1] 1788559646.115803 write_OD 0x26b000 thru 0x271fff (0x7000 bytes) 000203[1] 1788559646.181015 punch from 0x62c934 to 0x62e689 (0x1d56 bytes) 000204[0] 1788559646.212029 read 0x650000 thru 0x6531d8 (0x31d9 bytes) 000206[0] 1788559646.236584 write 0x90c000 thru 0x9125fd (0x65fe bytes) 000208[0] 1788559646.260060 read 0x3f6000 thru 0x3fdadf (0x7ae0 bytes) 000211[1] 1788559646.308362 mapwrite 0x4f4000 thru 0x4fb1da (0x71db bytes) 000212[1] 1788559646.390509 mapwrite 0x597000 thru 0x59fbe5 (0x8be6 bytes) 000217[0] 1788559646.460841 punch from 0x19053e to 0x19111d (0xbe0 bytes) 000218[1] 1788559646.483596 mapread 0x59e000 thru 0x5a47bb (0x67bc bytes) 000221[1] 1788559646.515384 read_OD 0x043000 thru 0x04bfff (0x9000 bytes) 000223[1] 1788559646.522412 trunc from 0x9125fe to 0x5e906c (0x329591 bytes) 000224[1] 1788559646.565730 mapread 0x0db000 thru 0x0ea75a (0xf75b bytes) truncating to largest ever: 0x9f7b8b 000225[0] 1788559646.624863 trunc from 0x5e906d to 0x9f7b8a (0x40eb1e bytes) 000226[1] 1788559646.695493 mapwrite 0x304000 thru 0x312fc7 (0xefc8 bytes) 000227[1] 1788559646.936305 write_OD 0x069000 thru 0x06bfff (0x3000 bytes) 000228[0] 1788559647.057732 trunc from 0x9f7b8b to 0x541c23 (0x4b5f67 bytes) 000229[0] 1788559647.155637 write 0x697000 thru 0x6a2b7a (0xbb7b bytes) 000230[0] 1788559647.169781 mapread 0x4ec000 thru 0x4fb97a (0xf97b bytes) 000231[0] 1788559647.199502 trunc from 0x6a2b7b to 0x5f913f (0xa9a3b bytes) 000232[0] 1788559647.247970 punch from 0x5b8f84 to 0x5c869e (0xf71b bytes) 000233[0] 1788559647.257530 mapread 0x589000 thru 0x58a545 (0x1546 bytes) 000236[0] 1788559647.366572 mapread 0x04a000 thru 0x04b6ec (0x16ed bytes) 000237[0] 1788559647.416337 mapread 0x2da000 thru 0x2dfe1a (0x5e1b bytes) 000238[1] 1788559647.441743 mapwrite 0x04c000 thru 0x054c4b (0x8c4c bytes) 000240[1] 1788559647.596035 read_OD 0x121000 thru 0x12efff (0xe000 bytes) 000242[0] 1788559647.637054 trunc from 0x5f9140 to 0x22d56e (0x3cbbd1 bytes) 000244[1] 1788559647.736568 write_OD 0x401000 thru 0x40efff (0xe000 bytes) 000245[1] 1788559647.795519 mapread 0x3e3000 thru 0x3ec2e2 (0x92e3 bytes) 000247[1] 1788559647.815362 punch from 0x15d710 to 0x15fa3f (0x2330 bytes) 000248[1] 1788559647.835669 write_OD 0x8ba000 thru 0x8befff (0x5000 bytes) 000249[0] 1788559647.889119 read 0x41b000 thru 0x42250a (0x750b bytes) 000250[0] 1788559647.900972 trunc from 0x8bf000 to 0x57af4b (0x3440b4 bytes) 000251[0] 1788559647.939581 trunc from 0x57af4c to 0x34df65 (0x22cfe6 bytes) 000252[0] 1788559647.972662 mapread 0x166000 thru 0x16e79d (0x879e bytes) 000254[0] 1788559648.018065 mapwrite 0x845000 thru 0x84b63a (0x663b bytes) 000256[1] 1788559648.173580 punch from 0x12334a to 0x128bd5 (0x588c bytes) 000258[0] 1788559648.203776 write 0x58f000 thru 0x5934f3 (0x44f4 bytes) 000262[1] 1788559648.274491 punch from 0x0ef658 to 0x0f1a21 (0x23ca bytes) 000267[0] 1788559648.302005 punch from 0x28c3da to 0x28e84c (0x2473 bytes) 000269[1] 1788559648.334049 punch from 0x6aaed8 to 0x6b35fc (0x8725 bytes) 000270[1] 1788559648.366671 write_OD 0x833000 thru 0x840fff (0xe000 bytes) 000273[1] 1788559648.431093 mapread 0x2b8000 thru 0x2bdb95 (0x5b96 bytes) 000274[1] 1788559648.446259 read_OD 0x02a000 thru 0x02cfff (0x3000 bytes) 000275[0] 1788559648.450794 read 0x259000 thru 0x263214 (0xa215 bytes) 000278[0] 1788559648.459406 trunc from 0x84b63b to 0x642a29 (0x208c11 bytes) 000279[0] 1788559648.517705 read 0x5f0000 thru 0x5fa3c3 (0xa3c4 bytes) 000282[0] 1788559648.559869 punch from 0x5bf905 to 0x5cf508 (0xfc04 bytes) 000285[1] 1788559648.564971 mapwrite 0x441000 thru 0x44d355 (0xc356 bytes) 000288[1] 1788559648.709411 mapwrite 0x829000 thru 0x832a60 (0x9a61 bytes) 000289[0] 1788559648.826700 punch from 0x471b48 to 0x48002f (0xe4e8 bytes) 000290[1] 1788559648.851595 mapread 0x520000 thru 0x528880 (0x8881 bytes) 000293[0] 1788559648.897484 punch from 0x62f9cf to 0x63b647 (0xbc79 bytes) 000294[0] 1788559648.927642 punch from 0x628d70 to 0x62c0ad (0x333e bytes) 000296[1] 1788559648.957390 mapread 0x05d000 thru 0x06a173 (0xd174 bytes) 000298[1] 1788559648.995021 mapread 0x526000 thru 0x535d05 (0xfd06 bytes) 000299[0] 1788559649.031129 mapwrite 0x0cd000 thru 0x0d07aa (0x37ab bytes) 000300[0] 1788559649.190217 write 0x3b7000 thru 0x3c39e9 (0xc9ea bytes) 000302[0] 1788559649.263992 mapread 0x119000 thru 0x123230 (0xa231 bytes) 000303[0] 1788559649.297891 read 0x34b000 thru 0x35350d (0x850e bytes) 000308[1] 1788559649.386424 trunc from 0x832a61 to 0x0c3303 (0x76f75d bytes) 000309[0] 1788559649.460659 read 0x08b000 thru 0x090ea5 (0x5ea6 bytes) 000310[0] 1788559649.522673 punch from 0x066d33 to 0x0721ee (0xb4bc bytes) 000312[1] 1788559649.579561 trunc from 0x0c3304 to 0x50cb94 (0x449891 bytes) 000313[0] 1788559649.604971 read 0x4bb000 thru 0x4c3cd9 (0x8cda bytes) 000314[1] 1788559649.708496 trunc from 0x50cb95 to 0x426d93 (0xe5e01 bytes) 000315[1] 1788559649.764559 mapread 0x223000 thru 0x22fbc5 (0xcbc6 bytes) 000316[0] 1788559649.805817 mapread 0x173000 thru 0x17d696 (0xa697 bytes) 000318[1] 1788559649.879367 punch from 0x1859db to 0x190c25 (0xb24b bytes) 000320[1] 1788559649.912481 write 0x6bd000 thru 0x6c24c6 (0x54c7 bytes) 000321[0] 1788559649.914785 punch from 0x638b4b to 0x647dd7 (0xf28d bytes) 000323[0] 1788559649.945474 write 0x8fa000 thru 0x8fbe83 (0x1e84 bytes) 000324[0] 1788559649.946826 read 0x2c6000 thru 0x2c80c6 (0x20c7 bytes) 000325[0] 1788559649.953535 mapread 0x28d000 thru 0x29673c (0x973d bytes) 000326[1] 1788559649.985774 write 0x9e7000 thru 0x9e87e3 (0x17e4 bytes) 000327[1] 1788559650.023457 read 0x95f000 thru 0x960844 (0x1845 bytes) 000329[0] 1788559650.065046 punch from 0x619cf7 to 0x620a8c (0x6d96 bytes) 000330[1] 1788559650.100818 read 0x661000 thru 0x66ff11 (0xef12 bytes) 000333[0] 1788559650.118501 write 0x269000 thru 0x26cb0c (0x3b0d bytes) 000334[1] 1788559650.150521 mapread 0x9b3000 thru 0x9bcbb7 (0x9bb8 bytes) 000335[0] 1788559650.219782 mapread 0x146000 thru 0x14f5ca (0x95cb bytes) 000337[0] 1788559650.236520 trunc from 0x9e87e4 to 0x391f04 (0x6568df bytes) 000340[0] 1788559650.275140 punch from 0x22c00d to 0x230e3a (0x4e2e bytes) 000342[0] 1788559650.282510 punch from 0x2528cb to 0x25e333 (0xba69 bytes) 000345[1] 1788559650.325452 mapwrite 0x596000 thru 0x59efbd (0x8fbe bytes) 000346[0] 1788559650.563405 punch from 0x0697bf to 0x079763 (0xffa5 bytes) 000349[0] 1788559650.611060 punch from 0x0f881a to 0x10741c (0xec03 bytes) 000350[0] 1788559650.616094 read 0x1f2000 thru 0x1fcca8 (0xaca9 bytes) 000352[0] 1788559650.640688 trunc from 0x59efbe to 0x500c2d (0x9e390 bytes) 000353[0] 1788559650.680935 mapread 0x46d000 thru 0x4765c8 (0x95c9 bytes) 000354[1] 1788559650.759644 mapwrite 0x729000 thru 0x72d81d (0x481e bytes) 000355[0] 1788559650.867546 mapread 0x294000 thru 0x29f370 (0xb371 bytes) 000356[0] 1788559650.922670 trunc from 0x72d81e to 0x007dd7 (0x725a46 bytes) 000357[0] 1788559650.970207 punch from 0x006dcc to 0x007dd7 (0x100c bytes) 000358[0] 1788559650.986778 trunc from 0x007dd8 to 0x2cd337 (0x2c5560 bytes) 000360[1] 1788559651.017779 mapwrite 0x452000 thru 0x45df36 (0xbf37 bytes) 000361[0] 1788559651.156474 mapread 0x23c000 thru 0x2425e5 (0x65e6 bytes) 000365[1] 1788559651.216125 read 0x12c000 thru 0x132c89 (0x6c8a bytes) 000366[0] 1788559651.232089 punch from 0x390706 to 0x39bf0f (0xb80a bytes) 000367[0] 1788559651.277517 mapwrite 0x9b2000 thru 0x9b9b7d (0x7b7e bytes) 000368[1] 1788559651.384766 trunc from 0x9b9b7e to 0x66215a (0x357a23 bytes) 000369[1] 1788559651.422367 mapread 0x117000 thru 0x126055 (0xf056 bytes) 000370[0] 1788559651.456405 read 0x548000 thru 0x557d9c (0xfd9d bytes) 000373[0] 1788559651.485104 trunc from 0x66215b to 0x021eed (0x64026d bytes) 000378[0] 1788559651.556396 mapwrite 0x5f8000 thru 0x605233 (0xd234 bytes) 000380[0] 1788559651.637578 punch from 0x37d822 to 0x384d26 (0x7505 bytes) 000383[1] 1788559651.657392 trunc from 0x605234 to 0x50850f (0xfcd24 bytes) 000384[1] 1788559651.718780 mapread 0x12e000 thru 0x12f264 (0x1265 bytes) 000388[0] 1788559651.741608 write 0x03d000 thru 0x0479eb (0xa9ec bytes) 000391[1] 1788559651.765958 read 0x142000 thru 0x147af4 (0x5af5 bytes) 000392[1] 1788559651.788824 trunc from 0x508510 to 0x5b8164 (0xafc55 bytes) 000393[0] 1788559651.833826 trunc from 0x5b8165 to 0x8aa30d (0x2f21a9 bytes) 000394[1] 1788559651.885574 punch from 0x11efa6 to 0x120de0 (0x1e3b bytes) 000396[1] 1788559651.928364 punch from 0x12b722 to 0x12dd3b (0x261a bytes) 000397[0] 1788559651.933270 read 0x66a000 thru 0x66fdb0 (0x5db1 bytes) 000398[0] 1788559651.953714 trunc from 0x8aa30e to 0x7c142d (0xe8ee0 bytes) 000399[0] 1788559651.997399 trunc from 0x7c142e to 0x5f2f5b (0x1ce4d2 bytes) 000400[1] 1788559652.037957 trunc from 0x5f2f5c to 0x722df7 (0x12fe9c bytes) 000401[1] 1788559652.058388 mapread 0x216000 thru 0x21a9cd (0x49ce bytes) 000403[0] 1788559652.081401 read 0x587000 thru 0x58daef (0x6af0 bytes) 000405[0] 1788559652.106045 punch from 0x245503 to 0x2513c0 (0xbebe bytes) 000406[0] 1788559652.155255 mapwrite 0x538000 thru 0x5461af (0xe1b0 bytes) 000408[1] 1788559652.236943 mapread 0x4ad000 thru 0x4aeef9 (0x1efa bytes) 000409[0] 1788559652.288115 mapwrite 0x7a1000 thru 0x7a4a01 (0x3a02 bytes) 000410[0] 1788559652.397401 mapwrite 0x1c1000 thru 0x1caf61 (0x9f62 bytes) 000413[0] 1788559652.509366 write 0x6c2000 thru 0x6c6786 (0x4787 bytes) 000415[0] 1788559652.542821 write 0x3c0000 thru 0x3cb33c (0xb33d bytes) 000416[0] 1788559652.559891 mapwrite 0x52e000 thru 0x538047 (0xa048 bytes) 000417[0] 1788559652.626554 write 0x3cb000 thru 0x3cc9a8 (0x19a9 bytes) 000418[0] 1788559652.644403 mapwrite 0x17c000 thru 0x18747e (0xb47f bytes) truncating to largest ever: 0x9f95b7 000419[1] 1788559652.717199 trunc from 0x7a4a02 to 0x9f95b6 (0x254bb5 bytes) 000420[0] 1788559652.746440 trunc from 0x9f95b7 to 0x1b8c07 (0x8409af bytes) 000423[0] 1788559652.801077 trunc from 0x1b8c08 to 0x5e3f34 (0x42b32d bytes) 000424[1] 1788559652.835221 mapread 0x11b000 thru 0x11dc91 (0x2c92 bytes) 000425[0] 1788559652.883993 punch from 0x1e8390 to 0x1f6a3e (0xe6af bytes) 000426[0] 1788559652.908717 trunc from 0x5e3f35 to 0x1852ec (0x45ec48 bytes) 000427[1] 1788559652.925469 write 0x4b1000 thru 0x4bc9d9 (0xb9da bytes) 000429[0] 1788559652.965611 trunc from 0x4bc9da to 0x7bc8c3 (0x2ffeea bytes) 000431[1] 1788559653.025606 mapwrite 0x351000 thru 0x355bae (0x4baf bytes) 000432[1] 1788559653.130591 write 0x8e8000 thru 0x8e949f (0x14a0 bytes) 000435[1] 1788559653.159208 punch from 0x6249dc to 0x62c961 (0x7f86 bytes) 000436[1] 1788559653.168408 read 0x5c1000 thru 0x5c3c9b (0x2c9c bytes) 000437[1] 1788559653.198824 read 0x17d000 thru 0x180b77 (0x3b78 bytes) 000438[0] 1788559653.211542 mapwrite 0x842000 thru 0x84eeb2 (0xceb3 bytes) 000440[1] 1788559653.329817 read 0x65e000 thru 0x660e00 (0x2e01 bytes) 000441[1] 1788559653.352459 write 0x275000 thru 0x27f15e (0xa15f bytes) 000442[1] 1788559653.373785 write 0x0ed000 thru 0x0f85e2 (0xb5e3 bytes) 000447[1] 1788559653.443052 trunc from 0x8e94a0 to 0x9e6747 (0xfd2a8 bytes) 000448[0] 1788559653.476693 read 0x686000 thru 0x68e637 (0x8638 bytes) 000449[0] 1788559653.550821 write 0x9cf000 thru 0x9d3cad (0x4cae bytes) 000450[1] 1788559653.603761 punch from 0x62de3e to 0x636d40 (0x8f03 bytes) 000451[1] 1788559653.639130 read 0x2ec000 thru 0x2f026f (0x4270 bytes) 000452[0] 1788559653.665164 read 0x34d000 thru 0x34f8b0 (0x28b1 bytes) 000454[0] 1788559653.680676 read 0x0a8000 thru 0x0af22b (0x722c bytes) 000458[1] 1788559653.688542 read 0x3d5000 thru 0x3e16d7 (0xc6d8 bytes) 000462[0] 1788559653.696536 punch from 0x0eed95 to 0x0efd48 (0xfb4 bytes) 000464[1] 1788559653.719684 punch from 0x3caf18 to 0x3d909b (0xe184 bytes) 000465[0] 1788559653.740791 mapwrite 0x1b7000 thru 0x1c5e39 (0xee3a bytes) 000466[0] 1788559653.843865 write 0x6a1000 thru 0x6a9b2b (0x8b2c bytes) 000468[1] 1788559653.850392 punch from 0x61fae0 to 0x62b462 (0xb983 bytes) 000469[1] 1788559653.872700 read 0x59e000 thru 0x5a5141 (0x7142 bytes) 000470[0] 1788559653.894187 trunc from 0x9e6748 to 0x4fc088 (0x4ea6bf bytes) 000472[0] 1788559653.942858 read 0x245000 thru 0x248e5e (0x3e5f bytes) 000473[1] 1788559653.960448 trunc from 0x4fc089 to 0x81449c (0x318414 bytes) 000474[1] 1788559653.988652 mapread 0x789000 thru 0x789af5 (0xaf6 bytes) 000476[1] 1788559654.031476 write 0x15b000 thru 0x16626a (0xb26b bytes) 000477[0] 1788559654.065272 punch from 0x7f596f to 0x7fb0c2 (0x5754 bytes) 000479[0] 1788559654.116610 mapwrite 0x3bb000 thru 0x3bce22 (0x1e23 bytes) 000480[1] 1788559654.175569 write 0x6ab000 thru 0x6ad3ef (0x23f0 bytes) 000481[1] 1788559654.198618 mapwrite 0x6f0000 thru 0x6f319c (0x319d bytes) 000482[0] 1788559654.299400 mapread 0x62c000 thru 0x635f5b (0x9f5c bytes) 000484[1] 1788559654.341429 trunc from 0x81449d to 0x67e26f (0x19622d bytes) 000485[1] 1788559654.372973 mapwrite 0x188000 thru 0x18d664 (0x5665 bytes) 000486[1] 1788559654.489058 read 0x43d000 thru 0x4497e3 (0xc7e4 bytes) 000487[0] 1788559654.497362 mapwrite 0x726000 thru 0x72e5f8 (0x85f9 bytes) 000489[0] 1788559654.590618 read 0x294000 thru 0x2a1ebc (0xdebd bytes) 000493[0] 1788559654.603846 mapwrite 0x46c000 thru 0x473ed9 (0x7eda bytes) 000494[0] 1788559654.684467 punch from 0x0b928e to 0x0c6a72 (0xd7e5 bytes) 000497[0] 1788559654.709785 punch from 0x72d102 to 0x72e5f8 (0x14f7 bytes) 000499[1] 1788559654.714731 mapwrite 0x62e000 thru 0x63d5dc (0xf5dd bytes) 000500[1] 1788559654.793319 trunc from 0x72e5f9 to 0x2ae019 (0x4805df bytes) All operations completed A-OK! PASS 16a (45s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 18:07:39 (1788559659) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.285859 s, 29.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.368073 s, 22.8 MB/s Chance of close/open is 1 in 50 Seed set to 9662 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003[1] 1788559663.068957 write 0x97ef0a thru 0x97f66d (0x764 bytes) 000004[1] 1788559663.107350 read 0x511167 thru 0x511678 (0x512 bytes) 000007[1] 1788559663.211186 read 0x386248 thru 0x3872fa (0x10b3 bytes) 000008[0] 1788559663.249847 mapwrite 0x3317bb thru 0x331a51 (0x297 bytes) 000010[1] 1788559663.379690 read 0x82b49d thru 0x82c6b9 (0x121d bytes) 000012[0] 1788559663.409949 mapwrite 0x3c419d thru 0x3c4d7d (0xbe1 bytes) 000014[0] 1788559663.510827 mapwrite 0x480e06 thru 0x48236c (0x1567 bytes) truncating to largest ever: 0x6d2a0e 000016[1] 1788559663.643102 trunc from 0x97f66e to 0x6d2a0d (0x2acc60 bytes) 000018[1] 1788559663.688168 mapread 0x0d5637 thru 0x0d5727 (0xf1 bytes) 000020[1] 1788559663.749741 trunc from 0x6d2a0e to 0x420a71 (0x2b1f9c bytes) 000022[0] 1788559663.805417 trunc from 0x420a72 to 0x5d02cc (0x1af85b bytes) 000025[0] 1788559663.928522 write 0x1f9a65 thru 0x1fa28f (0x82b bytes) 000026[1] 1788559663.974313 mapwrite 0x889de9 thru 0x88b309 (0x1521 bytes) 000028[0] 1788559664.188497 mapwrite 0x8282cc thru 0x8294c1 (0x11f6 bytes) 000029[1] 1788559664.303511 punch from 0x3e5320 to 0x3e5eda (0xbbb bytes) 000030[1] 1788559664.329122 punch from 0x437f75 to 0x439145 (0x11d1 bytes) 000031[0] 1788559664.357604 mapwrite 0x985aff thru 0x985b23 (0x25 bytes) 000034[1] 1788559664.497061 trunc from 0x985b24 to 0x0c64e8 (0x8bf63b bytes) 000037[1] 1788559664.612623 mapwrite 0x61a37b thru 0x61b759 (0x13df bytes) 000038[0] 1788559664.774867 write 0x35db96 thru 0x35eea6 (0x1311 bytes) 000039[1] 1788559664.810319 mapread 0x253387 thru 0x253df8 (0xa72 bytes) 000042[1] 1788559664.889738 mapread 0x051932 thru 0x0538cb (0x1f9a bytes) 000043[1] 1788559664.925751 write 0x332ea7 thru 0x33445d (0x15b7 bytes) 000044[0] 1788559664.973341 trunc from 0x61b75a to 0x5dadbd (0x4099c bytes) 000045[1] 1788559665.043151 read 0x3aedb7 thru 0x3aff92 (0x11dc bytes) truncating to largest ever: 0x74f7d0 000046[1] 1788559665.063389 trunc from 0x5dadbe to 0x74f7cf (0x174a12 bytes) 000047[0] 1788559665.100730 mapwrite 0x07b97c thru 0x07cd02 (0x1387 bytes) 000049[1] 1788559665.209555 read 0x62d9b2 thru 0x62eb7d (0x11cc bytes) 000050[1] 1788559665.228519 punch from 0x22ed2d to 0x22fbec (0xec0 bytes) 000051[1] 1788559665.253569 mapwrite 0x5d0bdc thru 0x5d17b7 (0xbdc bytes) 000052[1] 1788559665.372188 write 0x3d83c1 thru 0x3d84ea (0x12a bytes) 000054[0] 1788559665.412696 mapread 0x04bc3a thru 0x04d4a6 (0x186d bytes) 000055[0] 1788559665.486107 read 0x1c5526 thru 0x1c6869 (0x1344 bytes) 000056[1] 1788559665.495282 mapread 0x3abe92 thru 0x3acdd8 (0xf47 bytes) 000057[0] 1788559665.518102 write 0x0d27cd thru 0x0d44ff (0x1d33 bytes) 000058[0] 1788559665.562069 mapread 0x41d4e4 thru 0x41dc5a (0x777 bytes) 000059[0] 1788559665.584711 write 0x40b5ad thru 0x40c58f (0xfe3 bytes) 000060[0] 1788559665.605890 mapwrite 0x1d4b2f thru 0x1d56ef (0xbc1 bytes) 000062[0] 1788559665.656843 mapread 0x17e34a thru 0x17ec0b (0x8c2 bytes) 000063[1] 1788559665.669874 punch from 0x2426bb to 0x243448 (0xd8e bytes) 000065[1] 1788559665.710996 write 0x57aa32 thru 0x57bb87 (0x1156 bytes) 000066[1] 1788559665.757130 write 0x239798 thru 0x23a841 (0x10aa bytes) 000067[1] 1788559665.766722 write 0x4ef366 thru 0x4f09b8 (0x1653 bytes) 000068[0] 1788559665.784318 trunc from 0x74f7d0 to 0x4b6275 (0x29955a bytes) 000069[0] 1788559665.856158 mapread 0x0d9693 thru 0x0dab5b (0x14c9 bytes) 000072[0] 1788559665.870582 mapread 0x490376 thru 0x4903ad (0x38 bytes) 000074[0] 1788559665.905472 trunc from 0x4b6276 to 0x6ad739 (0x1f74c4 bytes) 000075[0] 1788559665.926691 mapread 0x39a06f thru 0x39a274 (0x206 bytes) 000077[0] 1788559665.966644 trunc from 0x6ad73a to 0x1ac684 (0x5010b5 bytes) 000080[1] 1788559666.063665 trunc from 0x1ac685 to 0x3f75e3 (0x24af5f bytes) 000082[0] 1788559666.139442 punch from 0x3e1f20 to 0x3e371d (0x17fe bytes) 000083[1] 1788559666.179054 mapread 0x3576a8 thru 0x3576d5 (0x2e bytes) 000084[1] 1788559666.254735 write 0x4a3dee thru 0x4a4446 (0x659 bytes) 000085[1] 1788559666.286020 mapread 0x33562a thru 0x3373af (0x1d86 bytes) 000086[0] 1788559666.374214 read 0x2a7150 thru 0x2a80a5 (0xf56 bytes) 000087[0] 1788559666.394064 punch from 0x468803 to 0x4692a5 (0xaa3 bytes) 000088[1] 1788559666.416950 read 0x48f2ab thru 0x48f8f7 (0x64d bytes) 000091[1] 1788559666.448934 mapread 0x1ca407 thru 0x1cbada (0x16d4 bytes) 000093[1] 1788559666.459335 read 0x385f63 thru 0x3876c2 (0x1760 bytes) 000097[1] 1788559666.465878 punch from 0x3c8180 to 0x3c9c28 (0x1aa9 bytes) 000098[1] 1788559666.493290 trunc from 0x4a4447 to 0x02d282 (0x4771c4 bytes) 000100[1] 1788559666.554634 write 0x69ca98 thru 0x69ea4e (0x1fb7 bytes) 000101[0] 1788559666.584632 mapread 0x042a58 thru 0x043c19 (0x11c2 bytes) 000102[1] 1788559666.653307 mapread 0x0647a2 thru 0x064996 (0x1f5 bytes) 000103[0] 1788559666.697224 mapwrite 0x8c63bf thru 0x8c7eac (0x1aee bytes) 000104[1] 1788559666.790651 trunc from 0x8c7ead to 0x6d2f3c (0x1f4f70 bytes) truncating to largest ever: 0x7a15e6 000105[1] 1788559666.850035 trunc from 0x6d2f3d to 0x7a15e5 (0xce6a9 bytes) truncating to largest ever: 0x8873d5 000108[0] 1788559666.971448 trunc from 0x7a15e6 to 0x8873d4 (0xe5def bytes) 000109[1] 1788559667.038124 read 0x12cd86 thru 0x12d7cb (0xa46 bytes) 000110[0] 1788559667.066269 mapread 0x34db98 thru 0x34f51e (0x1987 bytes) 000111[1] 1788559667.125865 mapread 0x30d9d5 thru 0x30e63e (0xc6a bytes) 000112[1] 1788559667.160306 mapwrite 0x139909 thru 0x13ab88 (0x1280 bytes) 000116[0] 1788559667.343573 write 0x5a4052 thru 0x5a40d1 (0x80 bytes) 000118[1] 1788559667.376282 mapwrite 0x3177b3 thru 0x317f08 (0x756 bytes) 000121[1] 1788559667.481210 read 0x69c4d7 thru 0x69cc53 (0x77d bytes) 000124[0] 1788559667.540137 trunc from 0x8873d5 to 0x725ac3 (0x161911 bytes) 000125[0] 1788559667.589887 mapread 0x367808 thru 0x368993 (0x118c bytes) 000127[1] 1788559667.600835 punch from 0x07066e to 0x07265f (0x1ff2 bytes) 000128[1] 1788559667.646740 punch from 0x0affed to 0x0b0c12 (0xc26 bytes) 000130[1] 1788559667.711699 write 0x7e16b1 thru 0x7e1953 (0x2a3 bytes) 000133[0] 1788559667.774308 trunc from 0x7e1954 to 0x263091 (0x57e8c2 bytes) 000134[0] 1788559667.821582 mapwrite 0x98d5f0 thru 0x98df8c (0x99d bytes) 000135[1] 1788559667.901715 mapread 0x09ca46 thru 0x09e487 (0x1a42 bytes) 000136[0] 1788559667.948112 write 0x7745b6 thru 0x775bcb (0x1616 bytes) 000137[0] 1788559667.964577 mapread 0x11002a thru 0x110ed3 (0xeaa bytes) 000138[0] 1788559667.969274 trunc from 0x98df8d to 0x73db31 (0x25045b bytes) 000139[1] 1788559667.992371 read 0x5c18b7 thru 0x5c1e62 (0x5ac bytes) truncating to largest ever: 0x94977b 000140[1] 1788559668.009472 trunc from 0x73db32 to 0x94977a (0x20bc49 bytes) 000141[0] 1788559668.109999 mapread 0x1a83fc thru 0x1a89f4 (0x5f9 bytes) 000143[0] 1788559668.166710 trunc from 0x94977b to 0x3e8e2f (0x56094b bytes) 000144[0] 1788559668.263843 read 0x1d710e thru 0x1d840f (0x1302 bytes) 000145[1] 1788559668.414198 write 0x90e30b thru 0x90ee50 (0xb46 bytes) 000147[0] 1788559668.469994 mapwrite 0x2d7a99 thru 0x2d9487 (0x19ef bytes) 000148[1] 1788559668.574498 mapread 0x75a1d2 thru 0x75a935 (0x764 bytes) 000150[1] 1788559668.618367 read 0x714a0e thru 0x714f1e (0x511 bytes) 000153[0] 1788559668.628748 mapread 0x7b6197 thru 0x7b7443 (0x12ad bytes) 000154[0] 1788559668.678355 read 0x6813f9 thru 0x682f51 (0x1b59 bytes) 000159[0] 1788559668.734874 trunc from 0x90ee51 to 0x7e505e (0x129df2 bytes) 000160[1] 1788559668.811412 punch from 0x234273 to 0x235db7 (0x1b45 bytes) 000164[1] 1788559668.935957 mapread 0x196b9e thru 0x198287 (0x16ea bytes) 000166[0] 1788559669.028442 punch from 0x3bc3d4 to 0x3bce23 (0xa50 bytes) 000167[0] 1788559669.051540 write 0x2b9caf thru 0x2bb09f (0x13f1 bytes) 000169[0] 1788559669.109415 mapwrite 0x5df7f8 thru 0x5e1530 (0x1d39 bytes) 000171[0] 1788559669.204045 write 0x21c21b thru 0x21e19d (0x1f83 bytes) 000173[0] 1788559669.259997 read 0x3f1b0c thru 0x3f2064 (0x559 bytes) 000175[1] 1788559669.310517 punch from 0x398a47 to 0x398d80 (0x33a bytes) 000176[1] 1788559669.368504 mapwrite 0x4a321a thru 0x4a489f (0x1686 bytes) 000177[1] 1788559669.501870 read 0x74617d thru 0x746ade (0x962 bytes) 000179[0] 1788559669.508984 punch from 0x611532 to 0x6126b2 (0x1181 bytes) 000180[1] 1788559669.560164 trunc from 0x7e505f to 0x29a42f (0x54ac2f bytes) 000181[1] 1788559669.603056 write 0x447814 thru 0x4492f6 (0x1ae3 bytes) 000182[0] 1788559669.605502 mapread 0x2cec63 thru 0x2cf6ac (0xa4a bytes) 000186[0] 1788559669.677540 mapwrite 0x4f4b5c thru 0x4f5703 (0xba8 bytes) 000189[1] 1788559669.807866 write 0x5b83b7 thru 0x5b8f54 (0xb9e bytes) 000191[0] 1788559669.869414 write 0x049071 thru 0x04adf6 (0x1d86 bytes) 000193[1] 1788559669.908410 write 0x9c5f6d thru 0x9c6ad3 (0xb67 bytes) 000194[0] 1788559669.930539 read 0x7ebd8f thru 0x7ec9ec (0xc5e bytes) 000195[1] 1788559669.984600 punch from 0x4a78e7 to 0x4a80be (0x7d8 bytes) 000196[0] 1788559670.026776 punch from 0x822f3e to 0x8248de (0x19a1 bytes) 000197[0] 1788559670.097495 mapread 0x650024 thru 0x65178e (0x176b bytes) 000199[0] 1788559670.139030 write 0x2bc362 thru 0x2bd215 (0xeb4 bytes) 000200[1] 1788559670.159782 trunc from 0x9c6ad4 to 0x5ceea3 (0x3f7c30 bytes) 000202[0] 1788559670.230835 punch from 0x00661d to 0x00757f (0xf63 bytes) 000204[1] 1788559670.275568 trunc from 0x5ceea4 to 0x7b6be0 (0x1e7d3d bytes) 000205[1] 1788559670.303450 mapread 0x5c420f thru 0x5c4a9c (0x88e bytes) 000207[1] 1788559670.326868 write 0x553bc4 thru 0x554243 (0x680 bytes) 000210[0] 1788559670.366071 read 0x3dd957 thru 0x3de252 (0x8fc bytes) 000215[0] 1788559670.409081 write 0x2f4f66 thru 0x2f6d16 (0x1db1 bytes) 000216[1] 1788559670.473190 trunc from 0x7b6be1 to 0x4c9c59 (0x2ecf87 bytes) 000221[0] 1788559670.689348 read 0x0af643 thru 0x0b0d03 (0x16c1 bytes) 000222[1] 1788559670.726337 punch from 0x41add1 to 0x41c874 (0x1aa4 bytes) truncating to largest ever: 0x98c952 000223[0] 1788559670.729771 trunc from 0x4c9c5a to 0x98c951 (0x4c2cf8 bytes) 000224[0] 1788559670.798551 mapread 0x74dccd thru 0x74e6e4 (0xa18 bytes) 000225[1] 1788559670.813542 mapwrite 0x1b7ef6 thru 0x1b9718 (0x1823 bytes) 000227[1] 1788559671.046231 mapread 0x277499 thru 0x278480 (0xfe8 bytes) 000228[1] 1788559671.081851 trunc from 0x98c952 to 0x1d2b2a (0x7b9e27 bytes) 000229[1] 1788559671.125936 trunc from 0x1d2b2b to 0x727226 (0x5546fc bytes) 000230[1] 1788559671.160162 mapread 0x22552c thru 0x225cca (0x79f bytes) 000231[0] 1788559671.173577 punch from 0x5025ca to 0x50424b (0x1c82 bytes) 000232[0] 1788559671.204955 punch from 0x05b32d to 0x05cb12 (0x17e6 bytes) 000234[1] 1788559671.219875 punch from 0x6051f1 to 0x6052e2 (0xf2 bytes) 000235[1] 1788559671.248808 write 0x766ff5 thru 0x768b2c (0x1b38 bytes) 000237[1] 1788559671.276518 mapwrite 0x629513 thru 0x62a6ee (0x11dc bytes) 000239[0] 1788559671.366403 write 0x212bdb thru 0x213b74 (0xf9a bytes) 000240[1] 1788559671.385684 punch from 0x41b932 to 0x41bdaa (0x479 bytes) 000245[1] 1788559671.454108 trunc from 0x768b2d to 0x29a86a (0x4ce2c2 bytes) 000246[0] 1788559671.514235 trunc from 0x29a86b to 0x5b689e (0x31c034 bytes) 000249[1] 1788559671.598676 mapwrite 0x14f04e thru 0x14f43f (0x3f2 bytes) 000250[1] 1788559671.732214 trunc from 0x5b689f to 0x49c725 (0x11a179 bytes) 000251[0] 1788559671.803649 trunc from 0x49c726 to 0x87f485 (0x3e2d60 bytes) 000253[1] 1788559671.912529 write 0x8a9ed3 thru 0x8aa573 (0x6a1 bytes) 000255[1] 1788559671.969567 mapread 0x1be42d thru 0x1be6da (0x2ae bytes) 000256[0] 1788559671.993724 mapread 0x216fcb thru 0x218f16 (0x1f4c bytes) 000257[0] 1788559672.033492 trunc from 0x8aa574 to 0x132d38 (0x77783b bytes) 000260[0] 1788559672.101514 write 0x1b7b98 thru 0x1b8d5d (0x11c6 bytes) 000262[0] 1788559672.123393 read 0x01db52 thru 0x01eb36 (0xfe5 bytes) 000264[1] 1788559672.193956 write 0x17ce10 thru 0x17d081 (0x272 bytes) 000267[0] 1788559672.249490 mapread 0x0463fd thru 0x048395 (0x1f99 bytes) 000268[1] 1788559672.294028 trunc from 0x1b8d5e to 0x652b2f (0x499dd2 bytes) 000269[0] 1788559672.357712 trunc from 0x652b30 to 0x2e0102 (0x372a2d bytes) 000270[1] 1788559672.424461 read 0x18ef5b thru 0x18fa80 (0xb26 bytes) 000271[1] 1788559672.458846 mapread 0x1653f0 thru 0x167001 (0x1c12 bytes) 000273[1] 1788559672.504378 read 0x0cc2ff thru 0x0ccd69 (0xa6b bytes) 000274[0] 1788559672.557286 punch from 0x11c2a8 to 0x11d08f (0xde8 bytes) 000275[0] 1788559672.613213 mapwrite 0x89a69b thru 0x89b2da (0xc40 bytes) 000276[1] 1788559672.829006 mapread 0x043ea8 thru 0x0458fe (0x1a57 bytes) 000278[1] 1788559672.948112 read 0x1210b7 thru 0x121d4b (0xc95 bytes) 000280[0] 1788559673.004431 trunc from 0x89b2db to 0x85a25c (0x4107e bytes) 000281[1] 1788559673.097821 write 0x81795f thru 0x817d03 (0x3a5 bytes) 000282[0] 1788559673.126374 punch from 0x68215d to 0x683d36 (0x1bda bytes) 000283[0] 1788559673.144303 read 0x2f76bf thru 0x2f7780 (0xc2 bytes) 000284[0] 1788559673.167213 trunc from 0x85a25d to 0x311c16 (0x548646 bytes) 000285[1] 1788559673.226101 punch from 0x2e0c6c to 0x2e24fe (0x1893 bytes) 000288[0] 1788559673.276007 trunc from 0x311c17 to 0x278886 (0x99390 bytes) 000289[1] 1788559673.321862 write 0x46a542 thru 0x46b08b (0xb4a bytes) 000290[0] 1788559673.351410 mapwrite 0x198bc9 thru 0x199edb (0x1313 bytes) 000293[0] 1788559673.432747 punch from 0x3ae2a8 to 0x3afa3a (0x1793 bytes) 000294[0] 1788559673.447212 trunc from 0x46b08c to 0x1ee9cb (0x27c6c0 bytes) 000296[0] 1788559673.480106 mapread 0x104357 thru 0x105614 (0x12be bytes) 000297[0] 1788559673.501658 punch from 0x14d0a9 to 0x14d75c (0x6b4 bytes) 000298[0] 1788559673.533040 trunc from 0x1ee9cc to 0x62033c (0x431971 bytes) 000299[1] 1788559673.563497 mapwrite 0x61ad5b thru 0x61be98 (0x113e bytes) 000300[1] 1788559673.658693 trunc from 0x62033d to 0x91b03a (0x2facfe bytes) 000301[1] 1788559673.708600 punch from 0x805604 to 0x806c3f (0x163c bytes) 000303[1] 1788559673.718267 mapread 0x51560b thru 0x516360 (0xd56 bytes) 000305[1] 1788559673.750306 mapwrite 0x435b44 thru 0x43722a (0x16e7 bytes) 000312[0] 1788559673.869089 write 0x56b159 thru 0x56c332 (0x11da bytes) 000313[1] 1788559673.903202 read 0x5f8a0f thru 0x5fa96e (0x1f60 bytes) 000314[0] 1788559673.935279 punch from 0x759797 to 0x759dca (0x634 bytes) 000315[1] 1788559673.959590 write 0x000205 thru 0x000ddc (0xbd8 bytes) 000316[0] 1788559673.974964 trunc from 0x91b03b to 0x4443cf (0x4d6c6b bytes) 000318[0] 1788559674.021096 punch from 0x2ed9f7 to 0x2ef315 (0x191f bytes) 000319[0] 1788559674.025344 trunc from 0x4443d0 to 0x988144 (0x543d75 bytes) 000320[0] 1788559674.038719 write 0x5af7c2 thru 0x5b132c (0x1b6b bytes) 000321[0] 1788559674.076856 read 0x259910 thru 0x25a6d8 (0xdc9 bytes) 000322[1] 1788559674.101793 trunc from 0x988145 to 0x4575d9 (0x530b6b bytes) 000323[1] 1788559674.155576 read 0x13e5ea thru 0x13f584 (0xf9b bytes) 000324[1] 1788559674.187593 write 0x176d6f thru 0x1787dc (0x1a6e bytes) 000326[0] 1788559674.200557 mapwrite 0x5d78c5 thru 0x5d882a (0xf66 bytes) 000327[1] 1788559674.297081 mapwrite 0x642c8e thru 0x643442 (0x7b5 bytes) 000328[0] 1788559674.365154 read 0x46e778 thru 0x46ee5d (0x6e6 bytes) 000331[1] 1788559674.422184 punch from 0x506887 to 0x507449 (0xbc3 bytes) 000333[1] 1788559674.465592 punch from 0x33ecd7 to 0x33f626 (0x950 bytes) 000334[0] 1788559674.469825 punch from 0x117fd6 to 0x118fe5 (0x1010 bytes) 000336[0] 1788559674.523781 mapwrite 0x699976 thru 0x69a54f (0xbda bytes) 000337[1] 1788559674.642837 write 0x967851 thru 0x968ece (0x167e bytes) 000338[1] 1788559674.676692 read 0x2a7ec1 thru 0x2a9337 (0x1477 bytes) 000339[1] 1788559674.701054 write 0x53b57b thru 0x53cd37 (0x17bd bytes) 000340[1] 1788559674.760162 write 0x4caac8 thru 0x4cbdcf (0x1308 bytes) 000343[0] 1788559674.820707 trunc from 0x968ecf to 0x67f185 (0x2e9d49 bytes) 000345[1] 1788559674.929764 punch from 0x2f7ae7 to 0x2f8505 (0xa1f bytes) 000346[1] 1788559674.978124 trunc from 0x67f186 to 0x2b357b (0x3cbc0a bytes) 000348[1] 1788559675.005126 trunc from 0x2b357c to 0x6fa0ad (0x446b32 bytes) 000349[1] 1788559675.038004 write 0x3d2811 thru 0x3d3633 (0xe23 bytes) 000350[0] 1788559675.069056 write 0x92d0f3 thru 0x92da8e (0x99c bytes) 000351[0] 1788559675.095606 trunc from 0x92da8f to 0x1fd28f (0x7307ff bytes) 000352[1] 1788559675.142287 write 0x4f531a thru 0x4f5f8f (0xc76 bytes) 000353[1] 1788559675.170376 read 0x02c0b3 thru 0x02d1bc (0x110a bytes) 000355[0] 1788559675.200630 write 0x5234d0 thru 0x524b5e (0x168f bytes) 000357[0] 1788559675.230005 mapwrite 0x680653 thru 0x68168f (0x103d bytes) 000358[0] 1788559675.318381 write 0x15f523 thru 0x16050a (0xfe8 bytes) 000359[1] 1788559675.341984 mapwrite 0x1b66ef thru 0x1b72cf (0xbe1 bytes) 000362[0] 1788559675.461153 trunc from 0x681690 to 0x478658 (0x209037 bytes) 000363[1] 1788559675.490360 mapread 0x047453 thru 0x047d82 (0x930 bytes) 000366[1] 1788559675.555570 read 0x240ba8 thru 0x24233e (0x1797 bytes) 000368[1] 1788559675.589672 punch from 0x1a232b to 0x1a40d7 (0x1dad bytes) 000369[1] 1788559675.616652 mapread 0x04e5f7 thru 0x050012 (0x1a1c bytes) 000370[1] 1788559675.670417 punch from 0x0e0421 to 0x0e1aff (0x16df bytes) 000371[1] 1788559675.686609 write 0x8fe56c thru 0x8ff2e2 (0xd77 bytes) 000372[1] 1788559675.710760 trunc from 0x8ff2e3 to 0x8d798c (0x27956 bytes) 000373[1] 1788559675.753077 read 0x4b33aa thru 0x4b36fa (0x351 bytes) 000374[1] 1788559675.757392 mapwrite 0x3c034d thru 0x3c04c1 (0x175 bytes) 000375[1] 1788559675.827143 mapread 0x711c64 thru 0x7125eb (0x988 bytes) 000376[0] 1788559675.884361 trunc from 0x8d798d to 0x6bf4c6 (0x2184c6 bytes) 000377[1] 1788559675.938643 trunc from 0x6bf4c7 to 0x8e76ad (0x2281e7 bytes) 000378[0] 1788559676.038708 read 0x3bc122 thru 0x3bd7a8 (0x1687 bytes) 000382[1] 1788559676.099458 trunc from 0x8e76ae to 0x7731d4 (0x1744d9 bytes) 000384[1] 1788559676.170224 punch from 0x197756 to 0x1990ff (0x19aa bytes) 000385[1] 1788559676.209498 mapread 0x5b2147 thru 0x5b2225 (0xdf bytes) 000386[1] 1788559676.214864 punch from 0x275dce to 0x277220 (0x1453 bytes) 000389[1] 1788559676.308368 mapread 0x3a4926 thru 0x3a501f (0x6fa bytes) 000391[0] 1788559676.364065 read 0x6f2c08 thru 0x6f479b (0x1b94 bytes) 000392[0] 1788559676.415687 read 0x3ef2e6 thru 0x3ef38d (0xa8 bytes) 000394[1] 1788559676.437746 trunc from 0x7731d5 to 0x2b44eb (0x4bece9 bytes) 000395[0] 1788559676.495497 mapread 0x09bd2b thru 0x09c1cd (0x4a3 bytes) 000396[1] 1788559676.541243 trunc from 0x2b44ec to 0x509c1e (0x255733 bytes) 000398[1] 1788559676.683881 punch from 0x4b6d1f to 0x4b6fbb (0x29d bytes) 000399[0] 1788559676.697734 read 0x4411a3 thru 0x44153f (0x39d bytes) 000401[1] 1788559676.724546 punch from 0x302744 to 0x303d4a (0x1607 bytes) 000402[0] 1788559676.751640 trunc from 0x509c1f to 0x354d57 (0x1b4ec7 bytes) 000403[1] 1788559676.826849 trunc from 0x354d58 to 0x87c736 (0x5279df bytes) 000405[0] 1788559676.919549 read 0x47a2d4 thru 0x47bd7f (0x1aac bytes) 000406[0] 1788559676.944577 trunc from 0x87c737 to 0x0b1486 (0x7cb2b0 bytes) 000408[1] 1788559677.022886 mapwrite 0x72311f thru 0x72398f (0x871 bytes) 000409[0] 1788559677.096078 trunc from 0x723990 to 0x307cd0 (0x41bcbf bytes) 000410[1] 1788559677.132501 write 0x0f8b15 thru 0x0f959b (0xa87 bytes) 000412[0] 1788559677.171475 mapwrite 0x4adc91 thru 0x4af1b7 (0x1527 bytes) 000413[1] 1788559677.295616 trunc from 0x4af1b8 to 0x7f8b0d (0x349956 bytes) 000414[0] 1788559677.355725 read 0x5e1d09 thru 0x5e20d0 (0x3c8 bytes) 000416[0] 1788559677.386001 trunc from 0x7f8b0e to 0x7bd86e (0x3b29f bytes) 000417[0] 1788559677.436006 punch from 0x0d7ed2 to 0x0d96e3 (0x1812 bytes) 000421[1] 1788559677.472891 mapwrite 0x919e87 thru 0x91a806 (0x980 bytes) 000423[0] 1788559677.761946 read 0x0371b9 thru 0x038de8 (0x1c30 bytes) 000424[0] 1788559677.805662 read 0x5f5f42 thru 0x5f6824 (0x8e3 bytes) 000425[1] 1788559677.835964 mapwrite 0x45610a thru 0x456332 (0x229 bytes) 000427[1] 1788559677.975641 trunc from 0x91a807 to 0x089199 (0x89166d bytes) 000428[0] 1788559678.040855 read 0x045d14 thru 0x046bde (0xecb bytes) 000429[1] 1788559678.060773 trunc from 0x08919a to 0x186b6c (0xfd9d3 bytes) 000431[1] 1788559678.118086 mapread 0x10c832 thru 0x10e139 (0x1908 bytes) 000432[0] 1788559678.157313 read 0x019cf7 thru 0x01ae6e (0x1178 bytes) 000433[0] 1788559678.204626 mapread 0x169a04 thru 0x16ad16 (0x1313 bytes) 000437[1] 1788559678.236296 mapread 0x02f53d thru 0x0304fe (0xfc2 bytes) 000438[1] 1788559678.257354 write 0x9f7789 thru 0x9f83f8 (0xc70 bytes) 000439[0] 1788559678.296390 mapread 0x7cf022 thru 0x7cf712 (0x6f1 bytes) 000440[1] 1788559678.341764 write 0x145aa2 thru 0x146c5a (0x11b9 bytes) 000441[0] 1788559678.364959 trunc from 0x9f83f9 to 0x436547 (0x5c1eb1 bytes) 000443[0] 1788559678.469615 read 0x2dfdee thru 0x2e01ed (0x400 bytes) 000444[0] 1788559678.493530 mapwrite 0x010155 thru 0x011e38 (0x1ce4 bytes) 000445[1] 1788559678.573619 write 0x621b8d thru 0x62330e (0x1782 bytes) 000447[1] 1788559678.612824 write 0x406c0f thru 0x407029 (0x41b bytes) 000449[0] 1788559678.650381 read 0x473b0d thru 0x4748fb (0xdef bytes) 000450[0] 1788559678.685437 read 0x0b4191 thru 0x0b504f (0xebf bytes) 000451[1] 1788559678.691845 mapwrite 0x9b2670 thru 0x9b3a84 (0x1415 bytes) 000452[1] 1788559678.815935 punch from 0x502ad3 to 0x504021 (0x154f bytes) 000453[1] 1788559678.839433 write 0x4bef13 thru 0x4c0e7c (0x1f6a bytes) 000457[0] 1788559678.875738 punch from 0x78cd45 to 0x78e8b3 (0x1b6f bytes) 000458[1] 1788559678.904567 mapread 0x177028 thru 0x177ec7 (0xea0 bytes) 000460[0] 1788559678.942540 write 0x924fa0 thru 0x926520 (0x1581 bytes) 000461[0] 1788559678.962778 trunc from 0x9b3a85 to 0x862f9c (0x150ae8 bytes) 000462[1] 1788559678.995751 write 0x2976a8 thru 0x29924c (0x1ba5 bytes) 000463[0] 1788559679.023792 write 0x6c56bd thru 0x6c6b1a (0x145e bytes) 000464[0] 1788559679.053190 trunc from 0x862f9d to 0x51a225 (0x348d77 bytes) 000465[1] 1788559679.084070 punch from 0x32b729 to 0x32ce3d (0x1715 bytes) 000466[0] 1788559679.117446 trunc from 0x51a226 to 0x5efe64 (0xd5c3f bytes) 000467[1] 1788559679.153797 mapwrite 0x124071 thru 0x124ddb (0xd6b bytes) 000468[1] 1788559679.249656 read 0x51fb7e thru 0x5209a5 (0xe28 bytes) 000470[1] 1788559679.284878 mapwrite 0x66151d thru 0x662277 (0xd5b bytes) 000474[0] 1788559679.468863 trunc from 0x662278 to 0x343ca6 (0x31e5d1 bytes) 000475[1] 1788559679.500409 mapread 0x333eba thru 0x335039 (0x1180 bytes) 000478[1] 1788559679.551242 read 0x19b49e thru 0x19c253 (0xdb6 bytes) 000479[0] 1788559679.636983 trunc from 0x343ca7 to 0x0b4051 (0x28fc55 bytes) 000480[1] 1788559679.704591 punch from 0x0536b8 to 0x05485d (0x11a6 bytes) 000481[0] 1788559679.735319 mapread 0x09a892 thru 0x09af3d (0x6ac bytes) 000482[0] 1788559679.783277 punch from 0x02f912 to 0x030bfe (0x12ed bytes) 000483[0] 1788559679.823443 read 0x01b728 thru 0x01cfac (0x1885 bytes) 000485[1] 1788559679.862333 read 0x08affe thru 0x08b5ef (0x5f2 bytes) 000486[1] 1788559679.921673 read 0x07e1a9 thru 0x08008e (0x1ee6 bytes) 000487[0] 1788559679.936875 write 0x934ab8 thru 0x935af2 (0x103b bytes) 000488[1] 1788559679.953471 punch from 0x407d34 to 0x40948b (0x1758 bytes) 000489[1] 1788559679.979049 write 0x90b414 thru 0x90be17 (0xa04 bytes) 000491[0] 1788559680.075292 mapread 0x4337e7 thru 0x4341e5 (0x9ff bytes) 000492[0] 1788559680.132341 write 0x44c16d thru 0x44cdc4 (0xc58 bytes) 000493[0] 1788559680.159710 punch from 0x356bd9 to 0x357253 (0x67b bytes) 000495[1] 1788559680.265271 punch from 0x7d95d5 to 0x7d9d43 (0x76f bytes) 000496[0] 1788559680.309229 read 0x6e3bef thru 0x6e4ee4 (0x12f6 bytes) 000499[0] 1788559680.345156 mapread 0x5fb91e thru 0x5fd766 (0x1e49 bytes) 000500[1] 1788559680.396368 punch from 0x0c363a to 0x0c42a4 (0xc6b bytes) All operations completed A-OK! PASS 16b (26s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 18:08:05 (1788559685) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 18:08:09 (1788559689) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.366078 s, 22.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.217782 s, 38.5 MB/s ldlm.namespaces.MGC192.168.206.124@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff99a64362d800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff99a6529e6800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff99a64362d800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff99a6529e6800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff99a64362d800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6724 s, 9.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.3352 s, 44.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.51148 s, 41.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.31489 s, 19.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.98256 s, 17.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.50181 s, 41.9 MB/s PASS 16d (40s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 18:08:49 (1788559729) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.290751 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879847 s, 11.9 MB/s PASS 16e (7s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 18:08:56 (1788559736) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 18:09:24 (1788559764) TIMEOUT PASS 16g (27s) == sanityn test 16h: mmap read after truncate file ======= 18:09:52 (1788559792) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (6s) == sanityn test 16i: read after truncate file ============ 18:09:57 (1788559797) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 18:10:05 (1788559805) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.23645 s, 21.2 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.96592 s, 13.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.01899 s, 6.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.46356 s, 10.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.10397 s, 8.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.69339 s, 9.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.46551 s, 7.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.50931 s, 17.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.83977 s, 14.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.19467 s, 8.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.53068 s, 7.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.96417 s, 13.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.10726 s, 12.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.63059 s, 10.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.71452 s, 9.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.03436 s, 8.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.50032 s, 7.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.5853 s, 7.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.67903 s, 7.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (42s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 18:10:48 (1788559848) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Fri Sep 4 18:11:33 EDT 2026 + fsx 2 done: rc=0 Fri Sep 4 18:11:33 EDT 2026 + fsx 3 done: rc=0 Fri Sep 4 18:11:33 EDT 2026 + fsx 4 done: rc=0 Fri Sep 4 18:11:33 EDT 2026 + fsx 5 done: rc=0 Fri Sep 4 18:11:33 EDT 2026 PASS 16k (55s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 18:11:42 (1788559902) fail_loc=0x8000030a PASS 17 (13s) == sanityn test 18: mmap sanity check =========================================================================================== 18:11:55 (1788559915) mmap test1: basic mmap operation (PASS, 0.14279s) mmap test2: MAP_PRIVATE not write back (PASS, 0.053317s) mmap test3: concurrent mmap ops on two nodes (PASS, 9.9853s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.464s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.084301s) mmap test6: check mmap write/read content on two nodes (PASS, 0.15215s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.71367s) mmap test11: multi-thread shared mmap (PASS, 0.60277s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1199s) PASS 18 (26s) == sanityn test 19: test concurrent uncached read races ========================================================================= 18:12:21 (1788559941) oleg624-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg624-client: oleg624-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (6s) == sanityn test 20: test extra readahead page left in cache ============================================================== 18:12:27 (1788559947) PASS 20 (10s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 18:12:37 (1788559957) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (10s) == sanityn test 23: others should see updated atime while another read============================================================== 18:12:48 (1788559968) now is 1788559971 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7641 new atime is 1788560032 PASS 23 (69s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 18:13:58 (1788560038) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (6s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 18:14:03 (1788560043) PASS 24b (6s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 18:14:09 (1788560049) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (6s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 18:14:16 (1788560056) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 18:14:19 (1788560059) utime: good mknod times 1788560059 <= 1788560060 <= 1788560060 for /mnt/lustre/f26a utime: good mknod times 1788560059 <= 1788560060 <= 1788560060 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (6s) == sanityn test 26b: sync mtime between ost and mds ====== 18:14:25 (1788560065) PASS 26b (7s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 18:14:33 (1788560073) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7641 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-04 18:14:34.000000000 -0400 Birth: 2026-09-04 18:14:34.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-04 18:14:34.000000000 -0400 Birth: 2026-09-04 18:14:34.000000000 -0400 PASS 26c (6s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 18:14:41 (1788560081) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 18:14:49 (1788560089) fail_loc=0x314 PASS 31a (9s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 18:14:58 (1788560098) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg624-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.206.124@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (22s) == sanityn test 31r: open-rename(replace) race =========== 18:15:21 (1788560121) -rw-r--r-- 1 root root 0 Sep 4 18:15 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (9s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 18:15:30 (1788560130) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43176 Terminated tail -f $tfile PASS 31s (10s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 18:15:40 (1788560140) PASS 31t (8s) == sanityn test 32b: lockless i/o ======================== 18:15:48 (1788560148) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (3s) == sanityn test 32c: contention detection ================ 18:15:51 (1788560151) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.44128 s, 36.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.27143 s, 41.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.95481 s, 26.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.08679 s, 25.1 MB/s contention_seconds 4, contention_events 18 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.32653 s, 22.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.39028 s, 21.9 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.24029 s, 42.3 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.25992 s, 41.6 MB/s PASS 32c (28s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 18:16:21 (1788560181) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 18:16:24 (1788560184) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 18:16:27 (1788560187) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (2s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 18:16:30 (1788560190) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 18:16:33 (1788560193) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg624-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff99a64362d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99a64362d800.ost_server_uuid in IDLE state after 0 sec oleg624-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99a64362d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99a64362d800.ost_server_uuid in FULL state after 0 sec oleg624-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff99a64362d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99a64362d800.ost_server_uuid in IDLE state after 0 sec oleg624-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99a64362d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99a64362d800.ost_server_uuid in FULL state after 0 sec oleg624-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff99a64362d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99a64362d800.ost_server_uuid in IDLE state after 0 sec oleg624-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99a64362d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99a64362d800.ost_server_uuid in FULL state after 0 sec PASS 34 (126s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 18:18:39 (1788560319) fail_loc=0x80000317 Race attempt 0 Wait for 50479 50531 for 60 sec... - open/close 838 (time 1788560335.45 total 10.01 last 83.69) - open/close 1639 (time 1788560345.45 total 20.01 last 80.10) - open/close 2484 (time 1788560355.45 total 30.02 last 84.49) - open/close 3317 (time 1788560365.45 total 40.02 last 83.26) total: 4000 open/close in 47.87 seconds: 83.56 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.206.124@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.206.124@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (72s) == sanityn test 36: handle ESTALE/open-unlink correctly == 18:19:51 (1788560391) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg624-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.943459 s, 55.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7641 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.33307 s, 22.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7641 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7491584) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.25007 s, 41.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7641 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7497728) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.952072 s, 55.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7641 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7493632) after_dd(7439360) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.14522 s, 45.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7641 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7497728) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.61679 s, 32.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7641 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7490560) after_dd(7442432) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.20749 s, 43.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7641 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.15063 s, 45.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7641 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.02631 s, 25.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7641 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7495680) after_dd(7440384) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.66114 s, 31.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7641 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.43247 s, 36.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7641 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(10) *** before(7497728) after_dd(7444480) after(7495680) PASS 36 (324s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 18:25:15 (1788560715) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7641 - create 1325 (time 1788560729.78 total 10.00 last 132.45) - create 2711 (time 1788560739.79 total 20.01 last 138.57) - create 4114 (time 1788560749.79 total 30.01 last 140.30) - create 5556 (time 1788560759.79 total 40.01 last 144.18) - create 7012 (time 1788560769.79 total 50.01 last 145.58) - create 8498 (time 1788560779.79 total 60.01 last 148.52) - create 9984 (time 1788560789.80 total 70.02 last 148.52) total: 10000 create in 70.15 seconds: 142.56 ops/second PASS 37 (81s) == sanityn test 39a: file mtime does not change after rename ========================================================== 18:26:36 (1788560796) repeat after cancel_lru_locks PASS 39a (6s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 18:26:43 (1788560803) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 18:26:51 (1788560811) repeat after cancel_lru_locks PASS 39c (8s) == sanityn test 39d: sync write should update mtime ====== 18:26:59 (1788560819) fail_loc=0x411 fail_loc=0 PASS 39d (6s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 18:27:08 (1788560828) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (23s) == sanityn test 40c: pdirops: link and others ======================================================================== 18:27:30 (1788560850) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (16s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 18:27:46 (1788560866) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (17s) == sanityn test 40e: pdirops: rename and others ======================================================================== 18:28:03 (1788560883) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (15s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 18:28:19 (1788560899) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (12s) == sanityn test 41b: pdirops: create vs create ======================================================================== 18:28:31 (1788560911) open: File exists Conflict PASS 41b (12s) == sanityn test 41c: pdirops: create vs link ======================================================================== 18:28:43 (1788560923) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (17s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 18:29:00 (1788560940) Conflict PASS 41d (14s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 18:29:15 (1788560955) Conflict PASS 41e (15s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 18:29:29 (1788560969) Conflict PASS 41f (14s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 18:29:43 (1788560983) Conflict PASS 41g (14s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 18:29:57 (1788560997) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (15s) == sanityn test 41i: reint_open: create vs create ======== 18:30:12 (1788561012) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1027s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 18:47:19 (1788562039) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (13s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 18:47:32 (1788562052) open: File exists Conflict PASS 42b (12s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 18:47:44 (1788562064) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (14s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 18:47:58 (1788562078) Conflict PASS 42d (13s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 18:48:11 (1788562091) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (14s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 18:48:25 (1788562105) Conflict PASS 42f (14s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 18:48:39 (1788562119) Conflict PASS 42g (13s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 18:48:52 (1788562132) Conflict PASS 42h (14s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 18:49:06 (1788562146) PASS 43a (68s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 18:50:14 (1788562214) Conflict PASS 43b (12s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 18:50:27 (1788562227) Conflict PASS 43c (12s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 18:50:39 (1788562239) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (13s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 18:50:52 (1788562252) Conflict PASS 43e (13s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 18:51:05 (1788562265) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (11s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 18:51:17 (1788562277) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (15s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 18:51:32 (1788562292) Conflict PASS 43h (17s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 18:51:49 (1788562309) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 18:51:52 (1788562312) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (115s) == sanityn test 43k: unlink vs create ==================== 18:53:47 (1788562427) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: ******unlink: No such file or directory sanityn test_43k: @@@@@@ FAIL: iter=6 : unlinking f43k.sanityn should succeed (err=2); Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7524:error() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:2885:test_43k() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7901:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7964:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7778:run_test() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:2895:main() Dumping lctl log to /tmp/testlogs//sanityn.test_43k.*.1788563062.log rsync: chown "/tmp/testlogs/.sanityn.test_43k.debug_log.oleg624-server.1788563062.log.rX6EBB" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanityn.test_43k.dmesg.oleg624-server.1788563062.log.AiRI0y" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 43k (639s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 19:04:33 (1788563073) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Sep 4 19:04:42 EDT 2026 PASS 44a (14s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 19:04:47 (1788563087) open: File exists Conflict PASS 44b (13s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 19:05:00 (1788563100) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (13s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 19:05:13 (1788563113) Conflict PASS 44d (13s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 19:05:26 (1788563126) Conflict PASS 44e (17s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 19:05:44 (1788563144) Conflict PASS 44f (14s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 19:05:59 (1788563159) Conflict PASS 44g (12s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 19:06:12 (1788563172) Conflict PASS 44h (13s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 19:06:25 (1788563185) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 19:06:29 (1788563189) PASS 45a (125s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 19:08:34 (1788563314) Conflict PASS 45b (11s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 19:08:45 (1788563325) Conflict PASS 45c (13s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 19:08:58 (1788563338) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (12s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 19:09:10 (1788563350) Conflict PASS 45e (13s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 19:09:23 (1788563363) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (14s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 19:09:37 (1788563377) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (13s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 19:09:50 (1788563390) Conflict PASS 45h (13s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 19:10:03 (1788563403) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 19:10:07 (1788563407) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1118s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 19:28:45 (1788564525) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (11s) == sanityn test 46b: pdirops: link vs create ======================================================================== 19:28:56 (1788564536) open: File exists Conflict PASS 46b (11s) == sanityn test 46c: pdirops: link vs link ======================================================================== 19:29:07 (1788564547) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (11s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 19:29:18 (1788564558) Conflict PASS 46d (12s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 19:29:30 (1788564570) Conflict PASS 46e (13s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 19:29:44 (1788564584) Conflict PASS 46f (12s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 19:29:56 (1788564596) Conflict PASS 46g (13s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 19:30:09 (1788564609) Conflict PASS 46h (12s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 19:30:21 (1788564621) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 19:30:24 (1788564624) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (3s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 19:30:27 (1788564627) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 19:30:30 (1788564630) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 19:30:33 (1788564633) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 19:30:36 (1788564636) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (4s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 19:30:40 (1788564640) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (4s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 19:30:44 (1788564644) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 19:30:47 (1788564647) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0251193 s, 408 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (12s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 19:30:59 (1788564659) 0+1 records in 0+1 records out 158 bytes copied, 0.0267479 s, 5.9 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (199041) - No such process PASS 51a (8s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 19:31:07 (1788564667) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.017905 s, 57.2 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998937 s, 10.5 MB/s 3145728 PASS 51b (19s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 19:31:26 (1788564686) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0206109 s, 49.7 kB/s PASS 51c (10s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 19:31:36 (1788564696) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379811 s, 27.6 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (6s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 19:31:42 (1788564702) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2406 0x966 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2406 0x966 0x240000400 read lease(1) has applied. PASS 51e (7s) == sanityn test 54: rename locking ======================= 19:31:49 (1788564709) llite.lustre-ffff99a64362d800.xattr_cache=1 llite.lustre-ffff99a6529e6800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272504920 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 19:31:51.000000000 -0400 Modify: 2026-09-04 19:31:51.000000000 -0400 Change: 2026-09-04 19:31:51.000000000 -0400 Birth: 2026-09-04 19:31:51.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272504923 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 19:31:57.000000000 -0400 Modify: 2026-09-04 19:31:58.000000000 -0400 Change: 2026-09-04 19:31:58.000000000 -0400 Birth: 2026-09-04 19:31:57.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272504926 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 19:32:03.000000000 -0400 Modify: 2026-09-04 19:32:03.000000000 -0400 Change: 2026-09-04 19:32:03.000000000 -0400 Birth: 2026-09-04 19:32:03.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272504925 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 19:32:03.000000000 -0400 Modify: 2026-09-04 19:32:04.000000000 -0400 Change: 2026-09-04 19:32:04.000000000 -0400 Birth: 2026-09-04 19:32:03.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272504928 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 19:32:09.000000000 -0400 Modify: 2026-09-04 19:32:10.000000000 -0400 Change: 2026-09-04 19:32:10.000000000 -0400 Birth: 2026-09-04 19:32:09.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272504929 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 19:32:09.000000000 -0400 Modify: 2026-09-04 19:32:10.000000000 -0400 Change: 2026-09-04 19:32:10.000000000 -0400 Birth: 2026-09-04 19:32:09.000000000 -0400 PASS 54 (30s) == sanityn test 55a: rename vs unlink target dir ========= 19:32:19 (1788564739) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (11s) == sanityn test 55b: rename vs unlink source dir ========= 19:32:30 (1788564750) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (11s) == sanityn test 55c: rename vs unlink orphan target dir == 19:32:41 (1788564761) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (17s) == sanityn test 55d: rename file vs link ================= 19:32:58 (1788564778) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (14s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 19:33:12 (1788564792) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (3s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 19:33:15 (1788564795) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (17s) == sanityn test 55g: rename: race with trylock =========== 19:33:32 (1788564812) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (20s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 19:33:52 (1788564832) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788564834 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.718107 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (94s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 19:35:26 (1788564926) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788564930 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.29866 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (105s) == sanityn test 60: Verify data_version behaviour ======== 19:37:11 (1788565031) PASS 60 (8s) == sanityn test 70a: cd directory && rm directory ======== 19:37:20 (1788565040) PASS 70a (9s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 19:37:28 (1788565048) PASS 70b (7s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 19:37:35 (1788565055) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.013776 s, 4.8 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 19:37:39 (1788565059) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0181272 s, 3.6 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 19:37:42 (1788565062) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 19:37:45 (1788565065) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0187277 s, 3.5 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (4s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 19:37:49 (1788565069) llite.lustre-ffff99a64362d800.xattr_cache=1 llite.lustre-ffff99a6529e6800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (7s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 19:37:56 (1788565076) llite.lustre-ffff99a64362d800.xattr_cache=1 llite.lustre-ffff99a6529e6800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (9s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 19:38:06 (1788565086) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0148161 s, 69.1 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0499768 s, 20.5 kB/s 211843: get lock1 211842: get lock2 211842: done 211842: get lock1 211843: done 211843: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 211843: cannot get lock2: Resource deadlock avoided 211843: exit rc=0 211842: done 211842: exit rc=0 PASS 74 (12s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 19:38:17 (1788565097) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0315282 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476854 s, 22.0 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 212425 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (10s) == sanityn test 76: Verify MDT open_files listing ======== 19:38:27 (1788565107) opening files: ........ closing files: ........ PASS 76 (59s) == sanityn test 77a: check FIFO NRS policy =============== 19:39:26 (1788565166) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.224351 s, 74.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475205 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689494 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043103 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704261 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923238 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900917 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814984 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807625 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160734 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162548 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13847 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124039 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13496 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034693 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152866 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17713 s, 5.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99a6529e6800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323063 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532891 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359925 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334055 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337061 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766516 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420486 s, 24.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701388 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424493 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647475 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450243 s, 23.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=2 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469414 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512702 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00409978 s, 256 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533195 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06163 s, 17.0 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (7s) == sanityn test 77b: check CRR-N NRS policy ============== 19:39:33 (1788565173) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.340644 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643703 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107425 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111159 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135134 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465027 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059121 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950329 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818698 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085518 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581552 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231117 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12742 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227889 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261723 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240155 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99a6529e6800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168016 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028897 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730488 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079247 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800094 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063125 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055088 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138435 s, 75.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064698 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188087 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452793 s, 23.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552395 s, 19.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966263 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748566 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242105 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00995171 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520863 s, 20.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=3 removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg624-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg624-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.453356 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099845 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135566 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15161 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131774 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246064 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266202 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232947 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25725 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296805 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295679 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369869 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28793 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277439 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286701 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30794 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301872 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99a6529e6800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00973231 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457061 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706207 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778221 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526297 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05545 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758448 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293322 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302162 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481908 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952956 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733263 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443206 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453746 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520817 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554324 s, 18.9 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (12s) == sanityn test 77c: check ORR NRS policy ================ 19:39:45 (1788565185) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.599693 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103878 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121982 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167509 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15195 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192983 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30379 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403635 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.485717 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.472997 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.496324 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.502206 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.501463 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.50845 s, 2.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99a6529e6800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.497225 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.594638 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.522502 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381729 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539323 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448246 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834319 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830561 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758673 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705841 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789066 s, 13.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074503 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682737 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852327 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713909 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898362 s, 11.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109051 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927255 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158389 s, 6.6 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg624-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg624-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.371085 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344289 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628914 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999011 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107634 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769791 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291248 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302116 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298014 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301245 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325372 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39475 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99a6529e6800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.440119 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.444725 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136101 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00196581 s, 533 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157425 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974972 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159292 s, 65.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179751 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.512764 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.527532 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00777512 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256193 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00566604 s, 185 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741385 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.565059 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199293 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183969 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587189 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638064 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860957 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129354 s, 81.1 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.520641 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896239 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895934 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104882 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102721 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127061 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25235 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265785 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350059 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350513 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277298 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360457 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420895 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.469188 s, 2.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.485889 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386613 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.606902 s, 1.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99a6529e6800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037108 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125017 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528305 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00990902 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00369714 s, 284 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00232176 s, 452 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892087 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18959 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182096 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147751 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054932 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150092 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482743 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387736 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377254 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334101 s, 31.4 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (18s) == sanityn test 77d: check TRR nrs policy ================ 19:40:03 (1788565203) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.404225 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168267 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197407 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360669 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339108 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45084 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374065 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.489943 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.525983 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417102 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.55587 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.5408 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.524674 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.532741 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437559 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.43837 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.68374 s, 1.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99a6529e6800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00249599 s, 420 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00249582 s, 420 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105218 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177564 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011815 s, 88.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106593 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26982 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00609442 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693571 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093879 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498293 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069547 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592471 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351682 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442645 s, 23.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508266 s, 20.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.95606 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13125 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179355 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272147 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256943 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306669 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320528 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291396 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279021 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401987 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365646 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.413305 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361725 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320054 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376556 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.435497 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.435982 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126637 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144139 s, 72.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137813 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132052 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348207 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847963 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643314 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149294 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133661 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129225 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891324 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992894 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142135 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143829 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152562 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174393 s, 6.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=10 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg624-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg624-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.501275 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557583 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558938 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109344 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192653 s, 5.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184545 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157139 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132988 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147581 s, 7.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99a6529e6800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194475 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175935 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217596 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196437 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186125 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261854 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246343 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268692 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99a6529e6800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157825 s, 66.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119712 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146304 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165854 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118568 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135319 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106966 s, 98.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110247 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990361 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803357 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744426 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680546 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064435 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630417 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521157 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438106 s, 23.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (18s) == sanityn test 77e: check TBF NID nrs policy ============ 19:40:21 (1788565221) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.206.24@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.487479 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103359 s, 10.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937892 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103379 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134185 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131006 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156678 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114327 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576589 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207911 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193989 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355611 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253765 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417637 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417724 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312721 s, 3.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.472735 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140053 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00572653 s, 183 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00445779 s, 235 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622826 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500422 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00723467 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168445 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778446 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977458 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618763 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317629 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701675 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142721 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716531 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727898 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126402 s, 83.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.355346 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692567 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699696 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689738 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068345 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108525 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141562 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161222 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16644 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185706 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152236 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223784 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188808 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180025 s, 5.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99a6529e6800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19078 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251125 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235712 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410515 s, 25.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783346 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617455 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919231 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155871 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110181 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833824 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896238 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118651 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099043 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147669 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102587 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108133 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134937 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114729 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116443 s, 9.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=1 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.316273 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286007 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278153 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100108 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12181 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1363 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180114 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178562 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17255 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166274 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183593 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19833 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200912 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203956 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199666 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179416 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222082 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99a6529e6800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00257947 s, 407 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100894 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112001 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836783 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798217 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608392 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579544 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939553 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498532 s, 21.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395162 s, 26.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458802 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379028 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332212 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024711 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289533 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194247 s, 54.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.285429 s, 58.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432083 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205477 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562257 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378526 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661538 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653852 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268838 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716632 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102901 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249124 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23973 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236502 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265272 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305087 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299771 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147445 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302479 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116603 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131517 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00820498 s, 128 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=8 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131625 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00383851 s, 273 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00954546 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019172 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00305224 s, 344 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130547 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075232 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00992028 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013417 s, 78.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00917663 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00442862 s, 237 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140846 s, 74.4 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg624-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg624-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (20s) == sanityn test 77f: check TBF JobID nrs policy ========== 19:40:41 (1788565241) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg624-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.634998 s, 26.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827457 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503955 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103766 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101891 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181831 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209344 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289049 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224073 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270697 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376594 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246246 s, 4.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326253 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.491325 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394328 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.500956 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.560778 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263374 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00437321 s, 240 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132403 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00443124 s, 237 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133601 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103854 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798822 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670267 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598043 s, 17.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509491 s, 20.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601411 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049002 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410507 s, 25.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307543 s, 34.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374287 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268288 s, 39.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg624-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.875618 s, 19.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107285 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135199 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10241 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157342 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518533 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523381 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115148 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234644 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769736 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13939 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196336 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191265 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105224 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880715 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281609 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181738 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981583 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100377 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513705 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074953 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137667 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378787 s, 27.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738325 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522797 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176533 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588536 s, 17.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173193 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904709 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151604 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776561 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830793 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844001 s, 12.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg624-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg624-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg624-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.10652 s, 15.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22572 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236499 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.395482 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277047 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226482 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290879 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400488 s, 2.6 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120569 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304691 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.521735 s, 2.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394281 s, 2.7 MB/s [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.770615 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281788 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.772892 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.674672 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.451861 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99a6529e6800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15681 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16097 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129928 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115797 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924518 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133479 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186725 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140161 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127162 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146245 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751442 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167455 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163121 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146317 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206795 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195948 s, 5.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg624-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg624-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg624-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.775017 s, 21.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745476 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108793 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171181 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928686 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331301 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187527 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100262 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399765 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736257 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115163 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380408 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209961 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.54974 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412489 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.6974 s, 1.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99a6529e6800.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 1+0 records in 1+0 records out [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.628464 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833517 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869824 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692425 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161342 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089574 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831856 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125854 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113453 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11538 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116139 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114847 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133992 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153085 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136278 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281801 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187969 s, 5.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (30s) == sanityn test 77g: Change TBF type directly ============ 19:41:12 (1788565272) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (13s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 19:41:26 (1788565286) oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg624-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg624-client: oleg624-server: ssh exited with exit code 2 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg624-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg624-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg624-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 PASS 77h (14s) == sanityn test 77i: Change rank of TBF rule ============= 19:41:40 (1788565300) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (24s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 19:42:05 (1788565325) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.567169 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949512 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185373 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123657 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317241 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353113 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200529 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348596 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253419 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364238 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210385 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366498 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.446509 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.474749 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.441102 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422465 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.477856 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99a6529e6800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339771 s, 3.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343632 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294592 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29456 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352815 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339734 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331861 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302178 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316343 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386424 s, 2.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348948 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331541 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303117 s, 3.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356404 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426714 s, 2.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349724 s, 3.0 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg624-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg624-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.1312 s, 8.0 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5888 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (75s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 19:43:20 (1788565400) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg624-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg624-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg624-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg624-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg624-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29638 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121917 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105928 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130719 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124232 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163187 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17753 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179746 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19642 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208362 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220504 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20652 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188114 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198674 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24045 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253619 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26889 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122383 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084899 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118123 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622953 s, 16.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120093 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697514 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151236 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167972 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918865 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930491 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077678 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154842 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953872 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135239 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100414 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121068 s, 8.7 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9533 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9945 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg624-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg624-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg624-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg624-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg624-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30294 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035795 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806536 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112487 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109457 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170279 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187575 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188484 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179451 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173213 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202111 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159081 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220507 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157104 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248618 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254533 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255548 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791916 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579073 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850029 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864211 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664566 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919708 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783897 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968088 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106868 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566923 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747242 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702334 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666638 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759784 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733361 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830087 s, 12.6 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg624-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg624-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.3634 s, 5.1 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9531 s, 5.3 MB/s Read runtime is 22 s, speed is 4.545454 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (152s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 19:45:52 (1788565552) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg624-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg624-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg624-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg624-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg624-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27555 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137493 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087184 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150643 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241246 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198973 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262889 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348511 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319018 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318759 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36988 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308113 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30877 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371322 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.453672 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.454129 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.523331 s, 2.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686007 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833383 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956864 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766437 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593387 s, 17.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05457 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980527 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989701 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107746 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635018 s, 16.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118674 s, 8.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972048 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103582 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15905 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1098 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145537 s, 7.2 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9872 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9572 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg624-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg624-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg624-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg624-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg624-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27611 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04574 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995653 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751946 s, 13.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107301 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115892 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110326 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135673 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995123 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138489 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124251 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153916 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124076 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164202 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187425 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139861 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198189 s, 5.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111422 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962897 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100144 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888928 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871151 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577092 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777424 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680389 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717468 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114472 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102956 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087211 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102593 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137224 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10545 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111063 s, 9.4 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg624-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg624-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9385 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.052 s, 5.2 MB/s Read runtime is 22 s, speed is 4.545454 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (156s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 19:48:28 (1788565708) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg624-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.65827 s, 25.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090508 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120982 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144809 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775721 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149527 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309542 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105315 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696826 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254263 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279244 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207223 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279098 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235976 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335557 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366999 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99a6529e6800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369676 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813525 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277224 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119754 s, 87.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167617 s, 62.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.007358 s, 143 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156254 s, 67.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192619 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0024322 s, 431 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207632 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189527 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709466 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115871 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896697 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122029 s, 85.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140536 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00946621 s, 111 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.79531 s, 13.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94314 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.206.24@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.206.24@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.980369 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100422 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123789 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115655 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183509 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285855 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281243 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280748 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25251 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220242 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26192 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215965 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22933 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308525 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282914 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225252 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99a6529e6800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297038 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114928 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11567 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124639 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155372 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129794 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160203 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131824 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13633 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1271 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125297 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131764 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141841 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13397 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147419 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141154 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158182 s, 6.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=2 removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg624-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg624-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.75227 s, 12.0 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93355 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.206.24@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg624-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.507829 s, 33.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895264 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110083 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138637 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152338 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158815 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108836 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14695 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164198 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19165 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208423 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220641 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222391 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184756 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178155 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181001 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189629 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027908 s, 37.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848486 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094999 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970904 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106923 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781884 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594919 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490628 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114685 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469512 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998292 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677299 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129334 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637953 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115404 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116435 s, 9.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg624-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg624-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3086 s, 10.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.27607 s, 19.9 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg624-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.620955 s, 27.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580233 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10709 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150036 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104026 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202441 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904256 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843947 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104877 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142319 s, 7.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216718 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142073 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23223 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109421 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145091 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164654 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189722 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731065 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837281 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861023 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854633 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628817 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687946 s, 15.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101876 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747805 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681614 s, 15.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798842 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785386 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848551 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816227 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921913 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12717 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119422 s, 8.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.6261 s, 8.3 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0029 s, 10.5 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.97572 s, 11.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0513 s, 10.4 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg624-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.05138 s, 16.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981875 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992917 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30354 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289397 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327046 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360704 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404395 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371954 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408534 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392049 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381967 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.427795 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471983 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.448059 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.488752 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.505667 s, 2.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704245 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140775 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109185 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872792 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672799 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651288 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913723 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122252 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927471 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101489 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888135 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111874 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110967 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1027 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112315 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12801 s, 8.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9505 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6616 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg624-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.333157 s, 50.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288593 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465121 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786336 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11261 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111427 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734392 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799367 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138202 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919821 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110327 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109594 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138227 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123833 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116208 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982043 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126167 s, 8.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323118 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345795 s, 30.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352326 s, 29.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353338 s, 29.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026599 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277619 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301537 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319951 s, 32.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329109 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378261 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345002 s, 30.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035367 s, 29.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408017 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373996 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340287 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375729 s, 27.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.37344 s, 16.5 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90982 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg624-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.860122 s, 19.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100962 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967345 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111818 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183122 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224351 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27705 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263675 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245812 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215231 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233538 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263789 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344098 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328156 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347081 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353167 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403055 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864601 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131957 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660399 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697812 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833979 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505285 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552969 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138414 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180493 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192426 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718614 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187425 s, 5.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122853 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19429 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100035 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102969 s, 10.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg624-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg624-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1124 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89633 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.1257 s, 8.6 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91911 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (439s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 19:55:47 (1788566147) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg624-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.614436 s, 27.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583667 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321014 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573769 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678254 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156486 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223742 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198657 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251274 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261216 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248684 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262906 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266352 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251959 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273094 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305312 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289802 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588766 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491913 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718725 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667777 s, 15.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117657 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106507 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668374 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762489 s, 13.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821437 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863266 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087127 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133003 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792187 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133877 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100118 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170934 s, 6.1 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg624-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg624-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.95272 s, 13.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9211 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (50s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 19:56:37 (1788566197) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.206.24@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.206.24@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.927699 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893159 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130765 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130346 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107486 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166943 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187967 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167431 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251421 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26936 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245744 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239209 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278686 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280864 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235963 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292173 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.483724 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654807 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101966 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112174 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130509 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113905 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170367 s, 61.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805912 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848576 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971072 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103475 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106118 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11402 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11927 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189806 s, 55.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194117 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125402 s, 8.4 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg624-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg624-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.26038 s, 11.3 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0043 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (58s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 19:57:35 (1788566255) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.206.24@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg624-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.590263 s, 28.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075706 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370972 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463715 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088546 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182132 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204152 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215752 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240751 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239089 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268815 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269605 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274192 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287341 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313053 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331567 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377509 s, 2.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff99a6529e6800.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582893 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050296 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122121 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057484 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599625 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701212 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656809 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191121 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754138 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845736 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917297 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213864 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117354 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110316 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115074 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145459 s, 7.2 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg624-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg624-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6298 s, 9.9 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.21216 s, 20.1 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (50s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 19:58:25 (1788566305) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg624-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.84342 s, 19.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565138 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843579 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083917 s, 12.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139654 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158462 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135141 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189704 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223089 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268525 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269744 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199664 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246143 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269203 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277251 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228761 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259806 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540209 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553581 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818507 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060788 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644998 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592633 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695573 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918805 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938681 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750369 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851491 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861153 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810445 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782522 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088492 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989798 s, 10.6 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg624-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg624-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4972 s, 10.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98292 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.68396 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88246 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (102s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 20:00:07 (1788566407) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg624-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.998051 s, 16.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807053 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836337 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182627 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156399 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113587 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201041 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21254 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192487 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194277 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291912 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261462 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168871 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21064 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260887 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137176 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224591 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898302 s, 11.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226402 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243805 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704805 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257167 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00745672 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742951 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800742 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606387 s, 17.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584942 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00431395 s, 243 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120366 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877152 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049329 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00633824 s, 165 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062031 s, 16.9 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1466 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8175 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (76s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 20:01:24 (1788566484) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg624-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.552315 s, 30.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314805 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548427 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627533 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06748 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457491 s, 22.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062267 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565498 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479606 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575268 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943179 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107143 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138522 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124395 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144146 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15933 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161498 s, 6.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056896 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622339 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865547 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718353 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802719 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13439 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967818 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827807 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102954 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106071 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106584 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115305 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120449 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131224 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136539 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156066 s, 6.7 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg624-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg624-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.62855 s, 10.9 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90874 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg624-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.795739 s, 21.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684392 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105274 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106787 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189095 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193417 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20916 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262329 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168897 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157187 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237159 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23929 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209594 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257074 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329009 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191045 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236526 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102335 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664446 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417329 s, 25.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441796 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413382 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534091 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506703 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835185 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153467 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824794 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860848 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962406 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103344 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102922 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107544 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121994 s, 8.6 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg624-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg624-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.2146 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90144 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6241 s, 9.9 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85387 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (151s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 20:03:55 (1788566635) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg624-client.virtnet /mnt/lustre (opts:) Stopping client oleg624-client.virtnet /mnt/lustre opts: Stopping clients: oleg624-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg624-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg624-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg624-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg624-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg624-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.24,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg624-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.24,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg624-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.24,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69588/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64-retry1 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.24,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg624-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg624-server' oleg624-server: oleg624-server.virtnet: executing load_modules_local oleg624-server: kmemleak disabled oleg624-server: Loading modules from /home/green/git/lustre-release/lustre oleg624-server: detected 4 online CPUs by sysfs oleg624-server: MODOPTS_LIBCFS= oleg624-server: Force libcfs to create 2 CPU partitions oleg624-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg624-server: oleg624-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg624-server: oleg624-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg624-server: oleg624-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre Starting client oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre Started clients oleg624-client.virtnet: 192.168.206.124@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre2 Starting client oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre2 Started clients oleg624-client.virtnet: 192.168.206.124@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff99a6436c3800.idle_timeout=debug osc.lustre-OST0000-osc-ffff99a649cc3000.idle_timeout=debug osc.lustre-OST0001-osc-ffff99a6436c3800.idle_timeout=debug osc.lustre-OST0001-osc-ffff99a649cc3000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.46817 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111348 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070753 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144977 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212649 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312145 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271121 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379852 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340279 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421304 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35735 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398793 s, 2.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41312 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332987 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417637 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.451329 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431615 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743354 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387976 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532256 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911754 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082516 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474259 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933838 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761474 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610465 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685414 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886416 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785272 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768146 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916086 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893477 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100281 s, 10.5 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg624-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg624-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.79496 s, 13.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.26707 s, 19.9 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.633239 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475396 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655262 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138554 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182483 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256068 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27765 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232377 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234473 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278589 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25921 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27272 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278582 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322191 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32636 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312037 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292344 s, 3.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff99a649cc3000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823574 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089257 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558269 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975724 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814091 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070059 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866577 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928349 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714172 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931975 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809317 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106577 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990277 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121222 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992773 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981339 s, 10.7 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.65398 s, 12.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88451 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (222s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 20:07:37 (1788566857) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg624-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (17s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 20:07:54 (1788566874) On MGS 192.168.206.124, active = nodemap.active=1 On MGS 192.168.206.124, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.619692 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871678 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889029 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099781 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143854 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293102 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301316 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292472 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296387 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237488 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247301 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286284 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266009 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23012 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317406 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308374 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99a649cc3000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff99a649cc3000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354469 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.535294 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.567994 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.534819 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.546133 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.558088 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.54988 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.557294 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.559291 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.588153 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.560085 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.564557 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.565597 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.562593 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.569674 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.584975 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.606521 s, 1.7 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.72703 s, 13.6 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.12311 s, 20.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.206.124, TBF.id = On MGS 192.168.206.124, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.667008 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648086 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962445 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801902 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799448 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121305 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138795 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192434 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190477 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292925 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287705 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228811 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99a649cc3000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17245 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229299 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251671 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233787 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383939 s, 2.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106017 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103416 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370787 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767225 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437221 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456294 s, 23.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99a649cc3000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff99a649cc3000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124408 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496627 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665573 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920248 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744546 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626773 s, 16.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655208 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676823 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862785 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705973 s, 14.9 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.89156 s, 15.2 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.85365 s, 36.7 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg624-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg624-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 On MGS 192.168.206.124, active = nodemap.active=1 On MGS 192.168.206.124, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.583642 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579866 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801403 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828941 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115888 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167586 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165184 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228996 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214772 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231185 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249409 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251081 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23315 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29977 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290933 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275353 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316244 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754404 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738233 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106064 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731358 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874589 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891874 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121603 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985563 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101233 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101961 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102165 s, 10.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102357 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105208 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107466 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104267 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123544 s, 8.5 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg624-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg624-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.98279 s, 11.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90548 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.206.124, TBF.id = On MGS 192.168.206.124, active = nodemap.active=0 PASS 77kj (170s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 20:10:44 (1788567044) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (8s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 20:10:52 (1788567052) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg624-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.60153 s, 291 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg624-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0966 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (56s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 20:11:49 (1788567109) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.333408 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577372 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576003 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311224 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317428 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283043 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261876 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726621 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161192 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188755 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130422 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114121 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168994 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241652 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225702 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251092 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182709 s, 5.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99a649cc3000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff99a649cc3000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474761 s, 22.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103521 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00514746 s, 204 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105148 s, 99.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102807 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713634 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636908 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116766 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658206 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106537 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122917 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991852 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071962 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119297 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868381 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794092 s, 13.2 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg624-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg624-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.18954 s, 11.4 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.35433 s, 19.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.65569 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393517 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620853 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950176 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133717 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140228 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16614 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172211 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195981 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154748 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160951 s, 6.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99a649cc3000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235543 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289858 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333068 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400752 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.395392 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.475847 s, 2.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff99a649cc3000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350045 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306039 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338092 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293059 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351327 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304002 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331095 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372294 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306504 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321851 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349027 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340608 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346079 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429331 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342837 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346819 s, 3.0 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg624-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg624-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.11205 s, 12.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.14344 s, 20.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (92s) == sanityn test 77o: Changing rank should not panic ====== 20:13:20 (1788567200) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (13s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 20:13:33 (1788567213) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (112s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 20:15:26 (1788567326) mds.MDS.mdt.nrs_policies=tbf oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg624-client: oleg624-server: ssh exited with exit code 36 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg624-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (37s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 20:16:03 (1788567363) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 254262 kill md_thread xargs PASS 77r (62s) == sanityn test 77s: Check TBF LRU shrinker ============== 20:17:05 (1788567425) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (64s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 20:18:09 (1788567489) oleg624-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg624-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg624-client: oleg624-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (9s) == sanityn test 79: xattr: intent error ================== 20:18:18 (1788567498) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935647 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 20:18:20.000000000 -0400 Modify: 2026-09-04 20:18:20.000000000 -0400 Change: 2026-09-04 20:18:20.000000000 -0400 Birth: 2026-09-04 20:18:20.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (18s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 20:18:36 (1788567516) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 20:18:39 (1788567519) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 20:18:42 (1788567522) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 20:18:46 (1788567526) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (2s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 20:18:49 (1788567529) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 20:18:52 (1788567532) - open/close 789 (time 1788567548.28 total 10.01 last 78.84) - open/close 1595 (time 1788567558.29 total 20.01 last 80.55) - open/close 2421 (time 1788567568.30 total 30.03 last 82.47) total: 2500 open/close in 30.95 seconds: 80.78 ops/second snapshot_time 1788567656.200875249 secs.nsecs start_time 1788567535.677506618 secs.nsecs elapsed_time 120.523368631 secs.nsecs open 2505 samples [usecs] 122 13423 3438893 8119609347 close 2504 samples [usecs] 130 6003 806165 503050289 mknod 2501 samples [usecs] 521 13318 2970996 6292104290 unlink 2500 samples [usecs] 694 57474 4744470 27417149852 mkdir 2 samples [usecs] 970 2246 3216 5985416 rmdir 2 samples [usecs] 11551 14056 25607 330996737 rename 2500 samples [usecs] 479 68633 16659412 262216965532 getattr 3762 samples [usecs] 10 7867 241413 343298949 parallel_rename_file 2500 samples [usecs] 479 68633 16659412 262216965532 crossdir_rename 2500 samples [usecs] 479 68633 16659412 262216965532 readpage 15 samples [usecs] 6738 30839 222366 4048110666 PASS 81d (128s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 20:21:01 (1788567661) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7641 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7641 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 20:21:07 (1788567667) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 20:21:11 (1788567671) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 20:21:24 (1788567684) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788567686 Finishing test test1 at 1788567690 PASS 85 (11s) == sanityn test 90: open/create and unlink striped directory ========================================================== 20:21:35 (1788567695) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 20:21:39 (1788567699) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 20:21:42 (1788567702) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 20:21:45 (1788567705) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5725 0x165d 0x240000400 1 5687 0x1637 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5688 0x1638 0x280000400 0 5726 0x165e 0x240000400 rc1=0 and rc2=0 PASS 93 (17s) == sanityn test 94: signal vs CP callback race =========== 20:22:02 (1788567722) osc.lustre-OST0000-osc-ffff99a6436c3800.idle_timeout=0 osc.lustre-OST0000-osc-ffff99a649cc3000.idle_timeout=0 osc.lustre-OST0001-osc-ffff99a6436c3800.idle_timeout=0 osc.lustre-OST0001-osc-ffff99a649cc3000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.328007 s, 25.0 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.00054 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 271275 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0777538 s, 52.7 kB/s osc.lustre-OST0000-osc-ffff99a6436c3800.idle_timeout=debug osc.lustre-OST0000-osc-ffff99a649cc3000.idle_timeout=debug osc.lustre-OST0001-osc-ffff99a6436c3800.idle_timeout=debug osc.lustre-OST0001-osc-ffff99a649cc3000.idle_timeout=debug PASS 94 (15s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 20:22:18 (1788567738) llite.lustre-ffff99a6436c3800.fast_read=0 llite.lustre-ffff99a649cc3000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0606058 s, 34.6 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff99a6436c3800.fast_read=1 llite.lustre-ffff99a649cc3000.fast_read=1 PASS 95a (21s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 20:22:38 (1788567758) llite.lustre-ffff99a6436c3800.fast_read=0 llite.lustre-ffff99a649cc3000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0100758 s, 1.2 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff99a6436c3800.fast_read=1 llite.lustre-ffff99a649cc3000.fast_read=1 PASS 95b (14s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 20:22:52 (1788567772) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 20:22:56 (1788567776) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00674202 s, 608 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (7s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 20:23:04 (1788567784) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.192329 s, 10.9 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (7s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 20:23:11 (1788567791) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.194468 s, 10.8 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (8s) == sanityn test 100e: DoM: read on open and file size ==== 20:23:20 (1788567800) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0112349 s, 1.1 MB/s PASS 100e (8s) == sanityn test 101a: Discard DoM data on unlink ========= 20:23:28 (1788567808) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0206501 s, 198 kB/s PASS 101a (9s) == sanityn test 101b: Discard DoM data on rename ========= 20:23:37 (1788567817) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00879358 s, 466 kB/s PASS 101b (8s) == sanityn test 101c: Discard DoM data on close-unlink === 20:23:45 (1788567825) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0192087 s, 213 kB/s PASS 101c (9s) == sanityn test 102a: Test open by handle of unlinked file ========================================================== 20:23:54 (1788567834) file: /mnt/lustre/f102a.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 40 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf40:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102a (8s) == sanityn test 102b: open by handle of a dir inside an encrypted dir ========================================================== 20:24:02 (1788567842) 192.168.206.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg624-client.virtnet /mnt/lustre2 (opts:) 192.168.206.124@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg624-client.virtnet /mnt/lustre (opts:) Starting client: oleg624-client.virtnet: -o user_xattr,flock,test_dummy_encryption 192.168.206.124@tcp:/lustre /mnt/lustre Starting client: oleg624-client.virtnet: -o user_xattr,flock,test_dummy_encryption 192.168.206.124@tcp:/lustre /mnt/lustre2 file: /mnt/lustre/d102b.sanityn/encdir/sub fh_bytes: 32 fh_type: 151 fh_data: d3 0b 00 00 02 00 00 00 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd3:0x3:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! 192.168.206.124@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,test_dummy_encryption=%s,test_dummy_encryption,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg624-client.virtnet /mnt/lustre (opts:) Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre 192.168.206.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,test_dummy_encryption=%s,test_dummy_encryption,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg624-client.virtnet /mnt/lustre2 (opts:) Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre2 PASS 102b (24s) == sanityn test 103: Test size correctness with lockahead ========================================================== 20:24:26 (1788567866) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788567870 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788567873 Finishing test test23 at 1788567873 PASS 103 (15s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 20:24:41 (1788567881) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 20:24:44 (1788567884) fail_loc=0x416 PASS 105 (20s) == sanityn test 106a: Verify the btime via statx() ======= 20:25:04 (1788567904) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 20:25:08 (1788567908) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0438003 s, 23.4 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 11 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238877265927 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 20:25:10.000000000 -0400 Modify: 2026-09-04 20:25:10.000000000 -0400 Change: 2026-09-04 20:25:10.000000000 -0400 Birth: 2026-09-04 20:25:10.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238877265927 -rw-r--r-- 1 0 0 1788567910 1788567910 1788567910 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 11 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238877265927 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 20:25:10.000000000 -0400 Modify: 2026-09-04 20:25:10.000000000 -0400 Change: 2026-09-04 20:25:10.000000000 -0400 Birth: 2026-09-04 20:25:10.000000000 -0400 1788567910 1024 11 PASS 106b (9s) == sanityn test 106c: Verify statx attributes mask ======= 20:25:17 (1788567917) PASS 106c (9s) == sanityn test 107a: Basic grouplock conflict =========== 20:25:26 (1788567926) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.266171 s, 39.4 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5695 0x163f 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7641 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7641 First grouplock blocks second one PASS 107a (9s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 20:25:36 (1788567936) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.371997 s, 28.2 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5735 0x1667 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7641 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (15s) == sanityn test 108a: lseek: parallel updates ============ 20:25:50 (1788567950) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0244673 s, 409 kB/s llite.lustre-ffff99a6480ab800.hybrid_io=0 llite.lustre-ffff99a6480af000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.68695 s, 1.8 MB/s PASS 108a (13s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 20:26:03 (1788567963) 192.168.206.124@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg624-client.virtnet /mnt/lustre (opts:) 192.168.206.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg624-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre2 192.168.206.124@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.206.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg624-client.virtnet /mnt/lustre (opts:) Stopping client oleg624-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg624-server' oleg624-server: oleg624-server.virtnet: executing load_modules_local oleg624-server: kmemleak disabled oleg624-server: Loading modules from /home/green/git/lustre-release/lustre oleg624-server: detected 4 online CPUs by sysfs oleg624-server: MODOPTS_LIBCFS= oleg624-server: Force libcfs to create 2 CPU partitions oleg624-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre2 Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre 192.168.206.124@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.206.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg624-client.virtnet /mnt/lustre (opts:) Stopping client oleg624-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg624-server' oleg624-server: oleg624-server.virtnet: executing load_modules_local oleg624-server: kmemleak disabled oleg624-server: Loading modules from /home/green/git/lustre-release/lustre oleg624-server: detected 4 online CPUs by sysfs oleg624-server: MODOPTS_LIBCFS= oleg624-server: Force libcfs to create 2 CPU partitions oleg624-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre2 PASS 109 (71s) == sanityn test 110: do not grant another lock on resend ========================================================== 20:27:14 (1788568034) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238910820355 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 20:27:16.000000000 -0400 Modify: 2026-09-04 20:27:16.000000000 -0400 Change: 2026-09-04 20:27:16.000000000 -0400 Birth: 2026-09-04 20:27:16.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg624-client.virtnet oleg624-client.virtnet PASS 110 (175s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 20:30:09 (1788568209) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (3s) == sanityn test 112: update max-inherit in default LMV === 20:30:12 (1788568212) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 20:30:16 (1788568216) lustre-OST0000-osc-ffff99a6480ad000 active. lustre-OST0001-osc-ffff99a6480ad000 active. lustre-OST0000-osc-ffff99a6480ad000 active. lustre-OST0001-osc-ffff99a6480ad000 active. MGC192.168.206.124@tcp active. PASS 113 (8s) == sanityn test 114: implicit default LMV inherit ======== 20:30:24 (1788568224) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (4s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 20:30:28 (1788568228) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 20:30:32 (1788568232) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (4s) == sanityn test 121: trunc append race =================== 20:30:35 (1788568235) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0107857 s, 1.9 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.110165 s, 186 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (9s) == sanityn test 122: directory size is consistent across mounts ========================================================== 20:30:44 (1788568244) PASS 122 (10s) == sanityn test 200: service remains healthy while able to process request ========================================================== 20:30:54 (1788568254) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1278 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.137 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.172 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1883 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2276 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2173 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2951 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1402 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1881 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2338 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1923 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1887 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2001 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3514 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.283 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3282 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2723 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3239 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg624-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff99a6480ad000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99a6480ad000.ost_server_uuid in FULL state after 0 sec PASS 200 (81s) cleanup: ====================================================== == sanityn test complete, duration 9143 sec ============== 20:32:17 (1788568337) sanityn: FAIL: test_43k iter=6 : unlinking f43k.sanityn should succeed (err=2); === sanityn: start cleanup 20:32:19 (1788568339) === Stopping clients: oleg624-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg624-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 20:36:25 (1788568585) ===