-----============= acceptance-small: sanityn ============----- Wed Sep 9 00:58:06 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg647-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 00:58:28 (1788929908) === Starting client oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre2 Mount client oleg647-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre2 Started clients oleg647-client.virtnet: 192.168.206.147@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg647-client.virtnet: executing check_config_client /mnt/lustre oleg647-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg647-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9004854a0000.idle_timeout=debug osc.lustre-OST0000-osc-ffff90049805d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9004854a0000.idle_timeout=debug osc.lustre-OST0001-osc-ffff90049805d000.idle_timeout=debug disable quota as required oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 00:59:07 (1788929947) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861885 s, 12.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7630] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 00:59:10 (1788929950) PASS 0 (11s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 00:59:21 (1788929961) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (8s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 00:59:29 (1788929969) -rw-r--r-- 1 root root 0 Sep 9 00:59 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 00:59:37 (1788929977) -rw-r--r-- 1 root root 0 Sep 9 00:59 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (8s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 00:59:45 (1788929985) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 00:59:52 (1788929992) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 00:59:59 (1788929999) total 1 -rwxrwxrwx 1 root root 0 Sep 9 00:59 f2a -rwxrwxrwx 1 root root 0 Sep 9 00:59 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 9 00:59 f2a -rwxrwxrwx 1 root root 0 Sep 9 00:59 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0178651 s, 28.7 kB/s PASS 2e (9s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 01:00:08 (1788930008) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 01:00:12 (1788930012) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.494954 s, 4.2 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (10s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 01:00:22 (1788930022) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 01:00:29 (1788930029) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 01:00:37 (1788930037) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 01:00:44 (1788930044) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (6s) == sanityn test 7: remove of open directory on other node ======================================================================= 01:00:50 (1788930050) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (7s) == sanityn test 8: remove of open special file on other node ==================================================================== 01:00:57 (1788930057) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (8s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 01:01:05 (1788930065) PASS 9a (8s) == sanityn test 9b: append to striped sparse file ======== 01:01:13 (1788930073) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681275 s, 15.4 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0483452 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (6s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 01:01:20 (1788930080) 1+0 records in 1+0 records out 1 byte copied, 0.00686464 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0102709 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0133294 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0176018 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0287205 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.025761 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0334785 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00924673 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.010925 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00922917 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0127666 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0161395 s, 0.1 kB/s PASS 10a (9s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 01:01:28 (1788930088) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0429692 s, 71.5 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0589456 s, 69.5 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00070764 s, 4.3 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 01:01:35 (1788930095) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (6s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 01:01:42 (1788930102) warning: '-runtime' deprecated, use '-t runtime' instead - link 358 (time 1788930114.65 total 10.03 last 35.68) - link 671 (time 1788930124.66 total 20.04 last 31.27) - link 997 (time 1788930134.68 total 30.06 last 32.54) - link 1339 (time 1788930144.69 total 40.08 last 34.15) - link 1661 (time 1788930154.73 total 50.11 last 32.08) - link 1939 (time 1788930164.76 total 60.15 last 27.70) - link 2235 (time 1788930174.79 total 70.18 last 29.51) - link 2574 (time 1788930184.80 total 80.19 last 33.88) - link 2897 (time 1788930194.82 total 90.21 last 32.23) total: 3222 link in 100.03 seconds: 32.21 ops/second using seed 3730156925 running for 100 seconds - stat 4602 (time 1788930116 ; total 11 ; last 11) - stat 9338 (time 1788930127 ; total 22 ; last 11) - stat 10000 (time 1788930128 ; total 23 ; last 1) - stat 14958 (time 1788930139 ; total 34 ; last 11) - stat 19804 (time 1788930150 ; total 45 ; last 11) - stat 20000 (time 1788930150 ; total 45 ; last 0) - stat 23677 (time 1788930161 ; total 56 ; last 11) - stat 25952 (time 1788930172 ; total 67 ; last 11) - stat 28703 (time 1788930183 ; total 78 ; last 11) - stat 30000 (time 1788930188 ; total 83 ; last 5) - stat 32319 (time 1788930199 ; total 94 ; last 11) total: 33848 stats in 100 seconds: 338.480011 stats/second - unlinked 0 (time 1788930155 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3222) error: No such file or directory total: 3221 unlinks in 86 seconds: 37.453487 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21455) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21457) - No such process PASS 12 (145s) == sanityn test 13: test directory page revocation ======= 01:04:06 (1788930246) f13 f13 PASS 13 (8s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 01:04:15 (1788930255) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 22640 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 01:04:21 (1788930261) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 23220 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 01:04:28 (1788930268) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23800 Terminated $DIR1/$tdir/sleep 60 PASS 14b (7s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 01:04:35 (1788930275) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24383 Terminated $DIR1/$tdir/sleep 60 PASS 14c (8s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 01:04:44 (1788930284) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24965 Terminated $DIR1/$tdir/sleep 60 PASS 14d (9s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 01:04:53 (1788930293) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (55s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 01:05:49 (1788930349) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.630078 s, 13.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.425818 s, 19.7 MB/s Chance of close/open is 1 in 50 Seed set to 354 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero length punch hole 000005[1] 1788930355.555334 mapwrite 0x1613db thru 0x162033 (0xc59 bytes) 000006[1] 1788930355.658291 write 0x61c122 thru 0x6233fa (0x72d9 bytes) 000007[1] 1788930355.666196 mapwrite 0x1600d4 thru 0x1688b2 (0x87df bytes) 000010[0] 1788930355.818102 read 0x15a324 thru 0x15c8b2 (0x258f bytes) 000011[1] 1788930355.848271 read 0x1dfc8c thru 0x1e7188 (0x74fd bytes) 000013[0] 1788930355.890479 punch from 0x5c4127 to 0x5c72eb (0x31c5 bytes) truncating to largest ever: 0x33745f 000014[1] 1788930355.929781 trunc from 0x6233fb to 0x33745e (0x2ebf9c bytes) 000015[1] 1788930355.994434 write 0x727c34 thru 0x72f459 (0x7826 bytes) 000016[1] 1788930356.027698 punch from 0x67499a to 0x67c43e (0x7aa5 bytes) 000017[1] 1788930356.053634 read 0x5d8555 thru 0x5e05be (0x806a bytes) 000021[1] 1788930356.150094 mapwrite 0x1a5c06 thru 0x1a87ea (0x2be5 bytes) 000022[0] 1788930356.298041 trunc from 0x72f45a to 0x26831d (0x4c713c bytes) 000026[0] 1788930356.499754 write 0x91dba4 thru 0x92c631 (0xea8e bytes) 000028[1] 1788930356.556211 mapread 0x59d8ef thru 0x5a2e9b (0x55ad bytes) 000029[1] 1788930356.654199 mapread 0x3a2fa0 thru 0x3a5441 (0x24a2 bytes) 000031[0] 1788930356.706208 read 0x3ca31c thru 0x3cdba0 (0x3885 bytes) 000033[0] 1788930356.713860 write 0x184c13 thru 0x189795 (0x4b83 bytes) 000034[0] 1788930356.758735 read 0x731ca6 thru 0x73a492 (0x87ed bytes) 000037[0] 1788930356.801876 trunc from 0x92c632 to 0x1ed462 (0x73f1cf bytes) 000038[1] 1788930356.879989 read 0x0575e4 thru 0x05cc80 (0x569d bytes) 000045[0] 1788930357.064117 read 0x09c03d thru 0x0a3e96 (0x7e5a bytes) 000047[0] 1788930357.170635 write 0x651da4 thru 0x652e1e (0x107b bytes) 000048[0] 1788930357.175403 write 0x7332fd thru 0x73b63c (0x8340 bytes) truncating to largest ever: 0x874c95 000049[0] 1788930357.181665 trunc from 0x73b63d to 0x874c94 (0x139658 bytes) 000051[0] 1788930357.260331 mapread 0x0ed51c thru 0x0f09b6 (0x349b bytes) 000053[1] 1788930357.355325 read 0x7a7b68 thru 0x7b3b1b (0xbfb4 bytes) 000054[0] 1788930357.403337 read 0x2355d4 thru 0x23ebef (0x961c bytes) 000055[0] 1788930357.416647 trunc from 0x874c95 to 0x3f3db4 (0x480ee0 bytes) 000056[0] 1788930357.449645 mapread 0x1b0bd5 thru 0x1b41a8 (0x35d4 bytes) 000058[1] 1788930357.467416 punch from 0x3bf3ac to 0x3c4a7a (0x56cf bytes) 000060[1] 1788930357.515105 mapwrite 0x37d059 thru 0x37d6f4 (0x69c bytes) 000063[0] 1788930357.608663 trunc from 0x3f3db5 to 0x79487c (0x3a0ac8 bytes) 000064[1] 1788930357.645145 trunc from 0x79487d to 0x63acac (0x159bd0 bytes) 000065[0] 1788930357.705252 punch from 0x40dc56 to 0x41845f (0xa80a bytes) 000066[1] 1788930357.757365 trunc from 0x63acad to 0x11422c (0x526a80 bytes) 000067[1] 1788930357.842741 read 0x0489f1 thru 0x05099b (0x7fab bytes) 000069[0] 1788930357.950475 mapwrite 0x6ba7e9 thru 0x6c795a (0xd172 bytes) 000071[1] 1788930358.133130 punch from 0x2727d5 to 0x276b54 (0x4380 bytes) 000073[0] 1788930358.191975 trunc from 0x6c795b to 0x4358f9 (0x292061 bytes) 000075[0] 1788930358.294290 punch from 0x291a91 to 0x29cbce (0xb13e bytes) 000081[1] 1788930358.344115 mapwrite 0x947cf2 thru 0x951971 (0x9c80 bytes) 000082[0] 1788930358.703819 punch from 0x5d0e28 to 0x5d5c1a (0x4df3 bytes) 000085[0] 1788930358.790440 mapwrite 0x46ec3f thru 0x474128 (0x54ea bytes) 000086[0] 1788930358.905805 write 0x67e3da thru 0x688361 (0x9f88 bytes) 000088[0] 1788930358.942134 read 0x4b6499 thru 0x4c2405 (0xbf6d bytes) 000089[1] 1788930358.965093 mapwrite 0x2fbd05 thru 0x3013b9 (0x56b5 bytes) 000090[1] 1788930359.039729 mapwrite 0x8802c2 thru 0x884d99 (0x4ad8 bytes) 000091[0] 1788930359.121899 mapwrite 0x5b9e0e thru 0x5bdf92 (0x4185 bytes) 000094[1] 1788930359.273670 mapread 0x493d21 thru 0x4991ae (0x548e bytes) 000095[1] 1788930359.331871 trunc from 0x951972 to 0x764cbc (0x1eccb5 bytes) 000096[1] 1788930359.393601 mapread 0x4978b0 thru 0x4a1a40 (0xa191 bytes) 000098[0] 1788930359.480880 read 0x246a2f thru 0x25196b (0xaf3d bytes) 000099[0] 1788930359.522770 mapread 0x3e8404 thru 0x3f092a (0x8527 bytes) 000100[1] 1788930359.562556 read 0x0dbc47 thru 0x0e6366 (0xa720 bytes) 000101[1] 1788930359.573468 mapread 0x4c7d6d thru 0x4d13fd (0x9691 bytes) 000102[0] 1788930359.599074 read 0x05418b thru 0x06380b (0xf681 bytes) 000105[1] 1788930359.726073 mapread 0x3ccca5 thru 0x3d0bc7 (0x3f23 bytes) 000108[0] 1788930359.789951 mapread 0x0fcfa1 thru 0x109983 (0xc9e3 bytes) 000110[0] 1788930359.802152 read 0x211eab thru 0x21870f (0x6865 bytes) 000112[1] 1788930359.822924 mapwrite 0x4c2f6c thru 0x4c7232 (0x42c7 bytes) 000113[1] 1788930359.929925 mapread 0x55a0d5 thru 0x55e2bb (0x41e7 bytes) 000116[0] 1788930360.020295 mapwrite 0x8caafd thru 0x8d93b9 (0xe8bd bytes) 000117[1] 1788930360.252440 write 0x186480 thru 0x18718f (0xd10 bytes) 000118[1] 1788930360.347705 read 0x56977a thru 0x5758f2 (0xc179 bytes) 000119[0] 1788930360.425062 read 0x5fdc5f thru 0x602b7d (0x4f1f bytes) 000124[0] 1788930360.477722 read 0x43532d thru 0x44466b (0xf33f bytes) 000125[1] 1788930360.605458 mapwrite 0x7c5ba7 thru 0x7cd2ab (0x7705 bytes) 000126[1] 1788930360.849547 mapwrite 0x5a424a thru 0x5af1d0 (0xaf87 bytes) 000129[0] 1788930361.023704 write 0x4f6897 thru 0x503702 (0xce6c bytes) 000131[1] 1788930361.084823 write 0x203175 thru 0x203299 (0x125 bytes) 000132[1] 1788930361.110096 write 0x341b7e thru 0x34faa0 (0xdf23 bytes) 000133[1] 1788930361.133030 write 0x4ff656 thru 0x50423a (0x4be5 bytes) 000135[0] 1788930361.174918 mapwrite 0x3f866b thru 0x3f8ba7 (0x53d bytes) 000136[1] 1788930361.414856 mapwrite 0x086d5f thru 0x094add (0xdd7f bytes) 000139[1] 1788930361.590040 write 0x3c5e4c thru 0x3cd3c0 (0x7575 bytes) 000140[0] 1788930361.634110 mapread 0x470ede thru 0x47f540 (0xe663 bytes) 000142[0] 1788930361.792956 mapread 0x74abe7 thru 0x758072 (0xd48c bytes) 000143[1] 1788930361.859899 punch from 0x6a20d7 to 0x6aeba8 (0xcad2 bytes) 000145[0] 1788930361.953400 punch from 0x1fba13 to 0x207552 (0xbb40 bytes) 000146[1] 1788930362.003756 mapread 0x07b66d thru 0x082e1a (0x77ae bytes) 000148[1] 1788930362.063244 mapread 0x1ab856 thru 0x1b938b (0xdb36 bytes) 000150[1] 1788930362.115162 write 0x144c0b thru 0x148da0 (0x4196 bytes) 000151[0] 1788930362.139461 write 0x724215 thru 0x72cc46 (0x8a32 bytes) 000153[0] 1788930362.206302 trunc from 0x8d93ba to 0x4db3d9 (0x3fdfe0 bytes) 000154[0] 1788930362.286495 punch from 0x124a15 to 0x12b1a5 (0x6791 bytes) 000164[1] 1788930362.529335 trunc from 0x4db3da to 0x229015 (0x2b23c4 bytes) 000165[0] 1788930362.592063 punch from 0x03fbb7 to 0x04af82 (0xb3cc bytes) 000167[1] 1788930362.647854 mapread 0x18308e thru 0x192e36 (0xfda9 bytes) 000168[1] 1788930362.706085 mapwrite 0x867dd8 thru 0x87329a (0xb4c3 bytes) 000170[1] 1788930362.832608 mapwrite 0x0f5392 thru 0x0fdd68 (0x89d7 bytes) 000171[1] 1788930362.910162 mapread 0x09eb98 thru 0x0a90d6 (0xa53f bytes) 000172[1] 1788930362.946266 trunc from 0x87329b to 0x043db8 (0x82f4e2 bytes) 000173[1] 1788930363.015365 trunc from 0x043db9 to 0x03906a (0xad4e bytes) 000174[0] 1788930363.081223 mapread 0x0349a9 thru 0x03906a (0x46c2 bytes) 000175[1] 1788930363.162691 write 0x4fc9b7 thru 0x5052f0 (0x893a bytes) 000176[0] 1788930363.199935 write 0x60f1ea thru 0x615602 (0x6419 bytes) 000177[0] 1788930363.296082 write 0x15d910 thru 0x15ece2 (0x13d3 bytes) 000183[1] 1788930363.521457 write 0x83f870 thru 0x84c9ff (0xd190 bytes) 000184[1] 1788930363.624072 mapwrite 0x4b225d thru 0x4bea41 (0xc7e5 bytes) 000186[0] 1788930363.846828 read 0x222c23 thru 0x227b09 (0x4ee7 bytes) 000187[0] 1788930363.897162 write 0x270a56 thru 0x27ccec (0xc297 bytes) 000188[0] 1788930363.946704 trunc from 0x84ca00 to 0x6fb935 (0x1510ca bytes) 000190[0] 1788930364.014131 mapread 0x6c28cf thru 0x6c94eb (0x6c1d bytes) 000191[1] 1788930364.059246 mapread 0x394679 thru 0x39fdff (0xb787 bytes) 000192[1] 1788930364.115458 read 0x4ac4c9 thru 0x4b0230 (0x3d68 bytes) 000193[0] 1788930364.148335 read 0x4445e3 thru 0x44c3fb (0x7e19 bytes) 000194[0] 1788930364.164487 mapread 0x01988d thru 0x0200b3 (0x6827 bytes) 000195[1] 1788930364.185570 punch from 0x199b3e to 0x19ad28 (0x11eb bytes) truncating to largest ever: 0x88fdfc 000196[1] 1788930364.230026 trunc from 0x6fb936 to 0x88fdfb (0x1944c6 bytes) 000199[1] 1788930364.290769 read 0x8200c1 thru 0x828879 (0x87b9 bytes) 000200[0] 1788930364.325054 write 0x3d6e5d thru 0x3dba20 (0x4bc4 bytes) 000201[1] 1788930364.365872 mapwrite 0x4ef1c9 thru 0x4f4b69 (0x59a1 bytes) 000202[1] 1788930364.479721 punch from 0x043915 to 0x0457a9 (0x1e95 bytes) 000203[0] 1788930364.515605 read 0x559ac4 thru 0x5687cd (0xed0a bytes) 000204[0] 1788930364.562554 write 0x6920fd thru 0x695054 (0x2f58 bytes) 000205[0] 1788930364.615205 mapread 0x2b1e48 thru 0x2b2bc7 (0xd80 bytes) 000206[1] 1788930364.649205 write 0x51c83d thru 0x51d01f (0x7e3 bytes) 000207[1] 1788930364.683327 punch from 0x154d5a to 0x1638f2 (0xeb99 bytes) 000208[1] 1788930364.726691 punch from 0x4ad0fc to 0x4ad489 (0x38e bytes) 000209[0] 1788930364.734836 write 0x54ac5b thru 0x553c14 (0x8fba bytes) 000210[0] 1788930364.773827 mapwrite 0x84b08c thru 0x852b2c (0x7aa1 bytes) 000211[1] 1788930364.922262 punch from 0x585215 to 0x586742 (0x152e bytes) 000212[1] 1788930365.013597 punch from 0x59f2bf to 0x5ad70b (0xe44d bytes) 000213[1] 1788930365.064364 mapwrite 0x3f511f thru 0x3fc8fe (0x77e0 bytes) 000214[1] 1788930365.219102 write 0x959fdf thru 0x9641eb (0xa20d bytes) 000216[0] 1788930365.292928 read 0x2cc486 thru 0x2d3bba (0x7735 bytes) 000217[0] 1788930365.332695 write 0x11ce35 thru 0x1229ba (0x5b86 bytes) 000219[0] 1788930365.406556 punch from 0x4689fe to 0x47712f (0xe732 bytes) 000220[1] 1788930365.458824 punch from 0x531eb6 to 0x53ee1b (0xcf66 bytes) 000221[0] 1788930365.494866 read 0x1c0ee4 thru 0x1d0545 (0xf662 bytes) 000222[0] 1788930365.531087 trunc from 0x9641ec to 0x043214 (0x920fd7 bytes) 000224[1] 1788930365.619721 write 0x25aabd thru 0x262982 (0x7ec6 bytes) 000225[0] 1788930365.660401 punch from 0x0b80f5 to 0x0c41f4 (0xc100 bytes) 000226[0] 1788930365.701723 trunc from 0x262983 to 0x2aefbd (0x4c63b bytes) 000227[0] 1788930365.773654 punch from 0x0573ce to 0x062e9a (0xbacd bytes) 000229[0] 1788930365.827581 write 0x8aaf91 thru 0x8ac822 (0x1892 bytes) 000230[1] 1788930365.885161 punch from 0x7116cc to 0x7124f4 (0xe29 bytes) 000231[0] 1788930365.963133 punch from 0x4d333a to 0x4d7407 (0x40ce bytes) 000232[0] 1788930365.983273 mapread 0x7e5373 thru 0x7eafdb (0x5c69 bytes) 000234[1] 1788930366.018321 write 0x7602d4 thru 0x76a92c (0xa659 bytes) 000239[1] 1788930366.179101 punch from 0x2a66bf to 0x2a84c6 (0x1e08 bytes) 000241[1] 1788930366.251263 write 0x8a45a3 thru 0x8a5aa8 (0x1506 bytes) 000242[0] 1788930366.318826 mapwrite 0x500799 thru 0x5078b9 (0x7121 bytes) 000243[0] 1788930366.573280 mapwrite 0x32008d thru 0x322c51 (0x2bc5 bytes) 000244[0] 1788930366.671269 write 0x21fcfb thru 0x22f6b6 (0xf9bc bytes) 000245[1] 1788930366.724272 punch from 0x0607d2 to 0x06eabc (0xe2eb bytes) 000248[0] 1788930366.816781 trunc from 0x8ac823 to 0x5eae80 (0x2c19a2 bytes) 000252[0] 1788930366.961341 trunc from 0x5eae81 to 0x59dc91 (0x4d1ef bytes) 000253[0] 1788930366.998983 read 0x0ca689 thru 0x0cf7d0 (0x5148 bytes) 000256[1] 1788930367.032926 mapread 0x595c04 thru 0x59dc91 (0x808e bytes) 000259[1] 1788930367.115277 punch from 0x36f678 to 0x3790f8 (0x9a81 bytes) 000260[0] 1788930367.154230 punch from 0x5880c3 to 0x597eae (0xfdec bytes) 000261[0] 1788930367.200771 write 0x28796b thru 0x2939b8 (0xc04e bytes) 000262[1] 1788930367.256646 mapread 0x2ad08b thru 0x2ba592 (0xd508 bytes) 000263[0] 1788930367.384696 write 0x69fefa thru 0x6a5b61 (0x5c68 bytes) 000264[0] 1788930367.386048 read 0x0c356d thru 0x0c890d (0x53a1 bytes) 000267[1] 1788930367.395231 write 0x6b5e16 thru 0x6c30cd (0xd2b8 bytes) 000268[1] 1788930367.437816 read 0x581639 thru 0x58df43 (0xc90b bytes) 000270[0] 1788930367.477801 mapread 0x2e01df thru 0x2e7749 (0x756b bytes) 000271[1] 1788930367.519092 punch from 0x32518e to 0x330a83 (0xb8f6 bytes) 000272[0] 1788930367.592433 mapread 0x5ec62e thru 0x5f3dfc (0x77cf bytes) 000273[1] 1788930367.673231 punch from 0x290924 to 0x29ab79 (0xa256 bytes) 000275[0] 1788930367.684416 write 0x7aded9 thru 0x7ba271 (0xc399 bytes) 000278[1] 1788930367.780184 mapread 0x6449db thru 0x6478f6 (0x2f1c bytes) 000279[0] 1788930367.870332 read 0x477e76 thru 0x47ffc2 (0x814d bytes) 000280[0] 1788930367.923612 trunc from 0x7ba272 to 0x496aa7 (0x3237ca bytes) 000281[1] 1788930367.972726 punch from 0x365646 to 0x367e81 (0x283c bytes) 000284[0] 1788930368.102074 trunc from 0x496aa8 to 0x0c3329 (0x3d377e bytes) 000286[1] 1788930368.211318 read 0x0b5aac thru 0x0c3329 (0xd87e bytes) 000287[1] 1788930368.265271 punch from 0x02d363 to 0x031f7b (0x4c19 bytes) 000288[1] 1788930368.308199 trunc from 0x0c332a to 0x0181e0 (0xab149 bytes) 000289[0] 1788930368.343225 mapread 0x00c608 thru 0x01049f (0x3e98 bytes) 000290[0] 1788930368.400305 write 0x14b2af thru 0x14df72 (0x2cc4 bytes) 000291[1] 1788930368.427447 read 0x13dcf2 thru 0x14af83 (0xd292 bytes) 000294[0] 1788930368.503811 read 0x027c3e thru 0x02f3cc (0x778f bytes) 000296[0] 1788930368.541851 write 0x47de40 thru 0x48a030 (0xc1f1 bytes) 000298[1] 1788930368.563084 read 0x3ce592 thru 0x3d28b3 (0x4322 bytes) 000299[0] 1788930368.602887 mapread 0x0c8996 thru 0x0d24f8 (0x9b63 bytes) 000300[1] 1788930368.618488 punch from 0x38b2a8 to 0x39a6a9 (0xf402 bytes) 000301[1] 1788930368.660752 write 0x9f55cc thru 0x9fffff (0xaa34 bytes) truncating to largest ever: 0x9cb861 000302[1] 1788930368.678032 trunc from 0xa00000 to 0x9cb860 (0x3479f bytes) 000305[1] 1788930368.752201 read 0x038663 thru 0x042b9d (0xa53b bytes) 000307[0] 1788930368.815364 read 0x7f0b9e thru 0x7fb2df (0xa742 bytes) 000308[1] 1788930368.848187 mapread 0x49d37b thru 0x4ac950 (0xf5d6 bytes) 000309[0] 1788930368.888890 punch from 0x473834 to 0x47ad9a (0x7567 bytes) 000310[0] 1788930368.920276 mapread 0x3fd427 thru 0x40c1bb (0xed95 bytes) 000311[1] 1788930368.976974 mapread 0x58fe48 thru 0x59dfa0 (0xe159 bytes) 000312[0] 1788930369.028967 mapwrite 0x575ea1 thru 0x57956c (0x36cc bytes) 000314[1] 1788930369.259863 read 0x220c81 thru 0x23009c (0xf41c bytes) 000317[0] 1788930369.315180 mapwrite 0x424758 thru 0x4278e9 (0x3192 bytes) 000318[0] 1788930369.404817 read 0x0b0ef1 thru 0x0bcf48 (0xc058 bytes) 000319[0] 1788930369.418190 read 0x075e15 thru 0x079ae7 (0x3cd3 bytes) 000320[1] 1788930369.438261 mapread 0x9c473a thru 0x9ca1be (0x5a85 bytes) 000324[1] 1788930369.535136 write 0x866c45 thru 0x86e500 (0x78bc bytes) 000326[1] 1788930369.671038 trunc from 0x9cb861 to 0x384d56 (0x646b0a bytes) 000329[0] 1788930369.750706 read 0x2b58d7 thru 0x2b9ac0 (0x41ea bytes) 000330[0] 1788930369.847322 punch from 0x2a63f2 to 0x2a95ec (0x31fb bytes) 000331[1] 1788930369.903995 read 0x06707c thru 0x0755b3 (0xe538 bytes) 000332[0] 1788930370.008335 mapread 0x156592 thru 0x15d416 (0x6e85 bytes) 000334[1] 1788930370.069335 trunc from 0x384d57 to 0x3a5732 (0x209dc bytes) 000335[1] 1788930370.126691 mapwrite 0x5158fb thru 0x524873 (0xef79 bytes) 000336[0] 1788930370.399127 mapread 0x23a05d thru 0x2422f3 (0x8297 bytes) 000338[1] 1788930370.506472 read 0x4c3be5 thru 0x4d3089 (0xf4a5 bytes) 000339[1] 1788930370.530783 mapread 0x2caf46 thru 0x2ce671 (0x372c bytes) 000342[0] 1788930370.661299 punch from 0x1e5d27 to 0x1f0b6f (0xae49 bytes) 000343[0] 1788930370.700307 punch from 0x4393e5 to 0x43b4ea (0x2106 bytes) 000345[1] 1788930370.724249 mapwrite 0x750bc7 thru 0x756f20 (0x635a bytes) 000346[0] 1788930370.839645 trunc from 0x756f21 to 0x07d56b (0x6d99b5 bytes) 000348[1] 1788930370.956859 mapread 0x06d0ff thru 0x077f61 (0xae63 bytes) 000350[1] 1788930371.058729 read 0x055461 thru 0x058ec2 (0x3a62 bytes) 000354[1] 1788930371.180910 write 0x97d5a2 thru 0x988699 (0xb0f8 bytes) 000355[0] 1788930371.238360 punch from 0x29d668 to 0x2a7e8d (0xa826 bytes) 000358[1] 1788930371.321861 mapwrite 0x9da214 thru 0x9e1ed0 (0x7cbd bytes) 000359[1] 1788930371.712744 write 0x707d96 thru 0x710b1e (0x8d89 bytes) 000361[1] 1788930371.719365 trunc from 0x9e1ed1 to 0x093b07 (0x94e3c9 bytes) 000363[0] 1788930371.835726 mapread 0x030526 thru 0x035fc4 (0x5a9f bytes) 000364[1] 1788930371.937944 write 0x900ccb thru 0x906c96 (0x5fcc bytes) 000365[1] 1788930371.958653 trunc from 0x906c97 to 0x7144b9 (0x1f27dd bytes) 000366[0] 1788930371.990317 punch from 0x6ad97c to 0x6b1551 (0x3bd6 bytes) 000367[1] 1788930372.041822 mapwrite 0x7aa72f thru 0x7b59fa (0xb2cc bytes) 000369[0] 1788930372.282622 read 0x073e6e thru 0x07871c (0x48af bytes) 000370[1] 1788930372.359532 read 0x612dc1 thru 0x61e9cf (0xbc0f bytes) 000371[0] 1788930372.383998 trunc from 0x7b59fb to 0x07f4ca (0x736530 bytes) 000373[1] 1788930372.475149 write 0x5c8254 thru 0x5d1acc (0x9879 bytes) 000374[0] 1788930372.514882 trunc from 0x5d1acd to 0x1a75a3 (0x42a529 bytes) 000375[0] 1788930372.603543 write 0x8cca33 thru 0x8d133d (0x490b bytes) 000376[0] 1788930372.638128 mapwrite 0x653e39 thru 0x65e446 (0xa60e bytes) 000377[0] 1788930372.785993 read 0x181e75 thru 0x18dcae (0xbe3a bytes) 000378[1] 1788930372.809159 mapread 0x41dff7 thru 0x425511 (0x751b bytes) 000379[0] 1788930372.889816 mapread 0x57c341 thru 0x589a37 (0xd6f7 bytes) 000381[0] 1788930372.949168 trunc from 0x8d133e to 0x76930d (0x168030 bytes) 000382[0] 1788930372.986661 trunc from 0x76930e to 0x15e8f5 (0x60aa18 bytes) 000385[0] 1788930373.061456 read 0x0fe8af thru 0x108879 (0x9fcb bytes) 000386[1] 1788930373.100428 write 0x97b417 thru 0x98335a (0x7f44 bytes) 000389[1] 1788930373.183068 punch from 0x920d08 to 0x926e2c (0x6125 bytes) 000390[1] 1788930373.232114 read 0x52d105 thru 0x52f7c4 (0x26c0 bytes) 000394[0] 1788930373.299208 mapwrite 0x782ee9 thru 0x785c64 (0x2d7c bytes) 000396[0] 1788930373.516497 trunc from 0x98335b to 0x618365 (0x36aff5 bytes) 000397[0] 1788930373.583119 write 0x839ac3 thru 0x8400a2 (0x65e0 bytes) 000399[0] 1788930373.664124 read 0x367d2e thru 0x36a5fe (0x28d1 bytes) 000401[0] 1788930373.686674 mapwrite 0x73ea15 thru 0x746be1 (0x81cd bytes) 000402[1] 1788930373.864131 mapread 0x4da6a9 thru 0x4e3849 (0x91a1 bytes) 000404[0] 1788930373.912934 mapread 0x0bcd8f thru 0x0c1d1b (0x4f8d bytes) 000405[0] 1788930373.955804 mapread 0x0111d5 thru 0x0168f4 (0x5720 bytes) 000408[0] 1788930373.998180 mapwrite 0x7b7ebb thru 0x7c3f53 (0xc099 bytes) 000409[0] 1788930374.239000 read 0x515cd4 thru 0x519b34 (0x3e61 bytes) 000414[1] 1788930374.302899 mapread 0x2e6e97 thru 0x2e97d3 (0x293d bytes) 000415[0] 1788930374.442749 trunc from 0x8400a3 to 0x24077c (0x5ff926 bytes) 000416[0] 1788930374.500414 write 0x4938e8 thru 0x49afde (0x76f7 bytes) 000417[0] 1788930374.541246 read 0x375334 thru 0x38093c (0xb609 bytes) 000420[1] 1788930374.665847 mapwrite 0x0e0139 thru 0x0e9e46 (0x9d0e bytes) 000422[1] 1788930374.795630 trunc from 0x49afdf to 0x69a000 (0x1ff022 bytes) 000427[0] 1788930374.940015 trunc from 0x69a001 to 0x841d78 (0x1a7d78 bytes) 000428[0] 1788930375.005791 trunc from 0x841d79 to 0x73a391 (0x1079e7 bytes) 000429[0] 1788930375.055038 punch from 0x476e6a to 0x482300 (0xb497 bytes) 000430[0] 1788930375.065050 trunc from 0x73a392 to 0x9c54dd (0x28b14c bytes) 000432[1] 1788930375.091680 write 0x9d77cc thru 0x9dc0f6 (0x492b bytes) 000434[1] 1788930375.159329 write 0x7901f8 thru 0x79dec0 (0xdcc9 bytes) 000437[1] 1788930375.203857 read 0x6ba60b thru 0x6c45ca (0x9fc0 bytes) 000438[0] 1788930375.213542 write 0x5d93e4 thru 0x5df2ff (0x5f1c bytes) 000440[1] 1788930375.272623 punch from 0x30e45e to 0x3197e6 (0xb389 bytes) 000441[0] 1788930375.294865 mapread 0x7612cf thru 0x76344d (0x217f bytes) 000442[0] 1788930375.348179 punch from 0x46f794 to 0x47c9d6 (0xd243 bytes) 000444[0] 1788930375.391811 read 0x8392fc thru 0x84813b (0xee40 bytes) 000445[1] 1788930375.421558 write 0x2a5adb thru 0x2ada70 (0x7f96 bytes) 000447[0] 1788930375.474285 write 0x8c77a3 thru 0x8c87bf (0x101d bytes) 000450[1] 1788930375.531615 mapwrite 0x4b4db9 thru 0x4b9905 (0x4b4d bytes) 000451[1] 1788930375.667610 write 0x03c680 thru 0x043ae1 (0x7462 bytes) 000452[1] 1788930375.712247 punch from 0x8f0295 to 0x8f34f7 (0x3263 bytes) 000453[1] 1788930375.745861 trunc from 0x9dc0f7 to 0x4a1c38 (0x53a4be bytes) 000455[1] 1788930375.817771 mapread 0x0b6fe1 thru 0x0bb94c (0x496c bytes) 000457[1] 1788930375.846363 punch from 0x20e519 to 0x2162b6 (0x7d9e bytes) 000458[1] 1788930375.869625 mapread 0x0a0061 thru 0x0a1b15 (0x1ab5 bytes) 000459[1] 1788930375.899589 write 0x19c3f7 thru 0x19e6d6 (0x22e0 bytes) 000460[1] 1788930375.940602 mapwrite 0x2c0de9 thru 0x2c7cbc (0x6ed4 bytes) 000461[1] 1788930376.055105 punch from 0x1bc3b1 to 0x1c245d (0x60ad bytes) 000462[1] 1788930376.091467 mapwrite 0x441481 thru 0x44fda1 (0xe921 bytes) 000463[1] 1788930376.233170 read 0x013b9f thru 0x013e2a (0x28c bytes) 000464[1] 1788930376.318060 mapwrite 0x98cac1 thru 0x99596d (0x8ead bytes) 000465[1] 1788930376.575283 punch from 0x8b0428 to 0x8b9856 (0x942f bytes) 000466[1] 1788930376.643160 write 0x324258 thru 0x329c8d (0x5a36 bytes) 000467[1] 1788930376.661580 read 0x35b678 thru 0x35dd5f (0x26e8 bytes) 000468[1] 1788930376.701848 mapread 0x0e8381 thru 0x0e9a0d (0x168d bytes) 000469[1] 1788930376.728293 mapread 0x7c8608 thru 0x7cddcc (0x57c5 bytes) 000470[1] 1788930376.835360 write 0x8e96c2 thru 0x8f1d47 (0x8686 bytes) 000472[1] 1788930376.902641 mapwrite 0x1908ba thru 0x19fec0 (0xf607 bytes) 000473[1] 1788930377.099551 mapread 0x5dcf2a thru 0x5e70d9 (0xa1b0 bytes) 000474[1] 1788930377.176695 trunc from 0x99596e to 0x3954c0 (0x6004ad bytes) 000475[0] 1788930377.232198 mapwrite 0x6d315d thru 0x6d9c78 (0x6b1c bytes) 000477[1] 1788930377.484088 mapread 0x46ea03 thru 0x474501 (0x5aff bytes) 000479[0] 1788930377.575680 write 0x1668ab thru 0x16ffb5 (0x970b bytes) 000480[1] 1788930377.607887 trunc from 0x6d9c79 to 0x180080 (0x559bf8 bytes) 000481[0] 1788930377.706296 write 0x7119bf thru 0x71996d (0x7faf bytes) 000483[0] 1788930377.795629 trunc from 0x71996e to 0x34753f (0x3d242e bytes) 000484[1] 1788930377.922949 read 0x03930b thru 0x03ffc6 (0x6cbc bytes) 000485[1] 1788930378.083488 read 0x31f570 thru 0x32981a (0xa2ab bytes) 000487[0] 1788930378.119645 read 0x05d830 thru 0x06d3ae (0xfb7f bytes) 000490[1] 1788930378.349946 mapread 0x21f755 thru 0x22eb63 (0xf40f bytes) 000491[0] 1788930378.392785 mapread 0x233fae thru 0x23743b (0x348e bytes) 000492[1] 1788930378.396630 write 0x1b7179 thru 0x1b71c0 (0x48 bytes) 000493[1] 1788930378.598295 punch from 0x31fb8d to 0x324e96 (0x530a bytes) 000494[1] 1788930378.635527 punch from 0x19a969 to 0x19f425 (0x4abd bytes) 000495[0] 1788930378.684158 mapread 0x18ecaa thru 0x196d1b (0x8072 bytes) 000496[1] 1788930378.877265 read 0x122aa7 thru 0x12c8ee (0x9e48 bytes) 000497[1] 1788930379.203324 punch from 0x184ef8 to 0x1933c8 (0xe4d1 bytes) 000498[0] 1788930379.277743 write 0x9c94c5 thru 0x9d0ce9 (0x7825 bytes) 000500[1] 1788930379.362218 write 0x7ae91f thru 0x7b67d9 (0x7ebb bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 379 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1788930380.599467 write 0x7b4000 thru 0x7bd8c9 (0x98ca bytes) truncating to largest ever: 0x3ab486 000002[0] 1788930380.636569 trunc from 0x7bd8ca to 0x3ab485 (0x412444 bytes) 000003[1] 1788930380.722530 mapread 0x122000 thru 0x12327e (0x127f bytes) 000004[0] 1788930380.781198 mapread 0x024000 thru 0x03291c (0xe91d bytes) truncating to largest ever: 0x71f970 000006[0] 1788930380.828507 trunc from 0x3ab486 to 0x71f96f (0x3744ea bytes) 000007[1] 1788930380.905245 mapread 0x520000 thru 0x52f31f (0xf320 bytes) 000013[0] 1788930380.999212 mapread 0x637000 thru 0x63d505 (0x6506 bytes) 000014[1] 1788930381.027268 punch from 0x0ee2c1 to 0x0f837d (0xa0bd bytes) truncating to largest ever: 0x78b212 000015[1] 1788930381.054094 trunc from 0x71f970 to 0x78b211 (0x6b8a2 bytes) 000016[0] 1788930381.092200 punch from 0x09ab31 to 0x0a6956 (0xbe26 bytes) 000018[0] 1788930381.144778 write 0x027000 thru 0x033f96 (0xcf97 bytes) 000019[0] 1788930381.152804 mapwrite 0x0c1000 thru 0x0c56bb (0x46bc bytes) 000022[0] 1788930381.350976 punch from 0x24b6b1 to 0x25a187 (0xead7 bytes) 000023[1] 1788930381.361764 trunc from 0x78b212 to 0x4b37de (0x2d7a33 bytes) 000024[1] 1788930381.422788 punch from 0x259e02 to 0x25e2dd (0x44dc bytes) 000025[0] 1788930381.430078 punch from 0x1b9b8f to 0x1bb03b (0x14ad bytes) 000026[1] 1788930381.462644 mapread 0x1c6000 thru 0x1ca3e2 (0x43e3 bytes) 000027[1] 1788930381.538325 write 0x06e000 thru 0x07b438 (0xd439 bytes) 000028[1] 1788930381.595153 mapread 0x113000 thru 0x11c252 (0x9253 bytes) 000030[0] 1788930381.660855 mapread 0x2b7000 thru 0x2c60a6 (0xf0a7 bytes) 000032[0] 1788930381.787552 write 0x358000 thru 0x35da51 (0x5a52 bytes) 000035[0] 1788930381.855273 mapwrite 0x290000 thru 0x2906d8 (0x6d9 bytes) 000039[0] 1788930381.997351 write 0x4c1000 thru 0x4c5cce (0x4ccf bytes) 000040[0] 1788930382.014162 write 0x0dc000 thru 0x0e0d3b (0x4d3c bytes) 000042[0] 1788930382.028077 read 0x4aa000 thru 0x4ad79a (0x379b bytes) 000044[1] 1788930382.059647 punch from 0x23583f to 0x239db3 (0x4575 bytes) 000045[1] 1788930382.113339 mapread 0x1de000 thru 0x1dfce9 (0x1cea bytes) 000047[0] 1788930382.166742 mapwrite 0x76c000 thru 0x76cd2a (0xd2b bytes) 000049[1] 1788930382.393138 mapread 0x69c000 thru 0x6a3757 (0x7758 bytes) 000050[1] 1788930382.546994 mapread 0x233000 thru 0x234409 (0x140a bytes) 000051[0] 1788930382.565279 trunc from 0x76cd2b to 0x30098c (0x46c39e bytes) truncating to largest ever: 0x87d9d1 000054[1] 1788930382.745926 trunc from 0x30098d to 0x87d9d0 (0x57d044 bytes) 000055[1] 1788930382.781837 mapwrite 0x847000 thru 0x84d534 (0x6535 bytes) 000056[1] 1788930382.905033 trunc from 0x87d9d1 to 0x72d6f0 (0x1502e0 bytes) 000057[1] 1788930382.939951 write 0x9d1000 thru 0x9d481d (0x381e bytes) truncating to largest ever: 0x967b7a 000058[0] 1788930382.957845 trunc from 0x9d481e to 0x967b79 (0x6cca4 bytes) 000059[1] 1788930383.058564 read 0x866000 thru 0x86cdb9 (0x6dba bytes) 000060[1] 1788930383.105104 read 0x687000 thru 0x695a9f (0xeaa0 bytes) 000062[1] 1788930383.132102 trunc from 0x967b7a to 0x10b997 (0x85c1e2 bytes) 000063[1] 1788930383.176289 write 0x0c7000 thru 0x0d6e34 (0xfe35 bytes) 000065[1] 1788930383.186407 punch from 0x0b7ed2 to 0x0c0849 (0x8978 bytes) 000072[0] 1788930383.298924 trunc from 0x10b998 to 0x2f1b6a (0x1e61d3 bytes) 000077[1] 1788930383.402942 mapwrite 0x9fd000 thru 0x9ff5af (0x25b0 bytes) 000078[1] 1788930383.555334 write 0x195000 thru 0x19a75a (0x575b bytes) 000079[0] 1788930383.574155 trunc from 0x9ff5b0 to 0x863de5 (0x19b7ca bytes) 000080[1] 1788930383.644202 mapread 0x0d1000 thru 0x0db2ab (0xa2ac bytes) 000081[0] 1788930383.715286 read 0x7e8000 thru 0x7ea193 (0x2194 bytes) 000082[1] 1788930383.721181 mapread 0x44b000 thru 0x44d8a3 (0x28a4 bytes) 000083[0] 1788930383.756788 trunc from 0x863de6 to 0x782890 (0xe1555 bytes) 000084[1] 1788930383.807255 trunc from 0x782891 to 0x610386 (0x17250a bytes) 000085[0] 1788930383.855815 mapread 0x2d1000 thru 0x2d2674 (0x1675 bytes) 000087[1] 1788930383.925072 read 0x542000 thru 0x545ab8 (0x3ab9 bytes) 000091[1] 1788930383.946124 mapwrite 0x453000 thru 0x454f1b (0x1f1c bytes) 000093[0] 1788930384.057426 punch from 0x5a12f5 to 0x5a376e (0x247a bytes) 000095[0] 1788930384.083769 trunc from 0x610387 to 0x759390 (0x14900a bytes) 000096[1] 1788930384.110740 punch from 0x62525e to 0x631d6a (0xcb0d bytes) 000097[1] 1788930384.149330 trunc from 0x759391 to 0x8d02b2 (0x176f22 bytes) 000098[1] 1788930384.183136 punch from 0x7d638b to 0x7e03e8 (0xa05e bytes) 000099[0] 1788930384.215029 punch from 0x39d086 to 0x3a79a2 (0xa91d bytes) 000100[1] 1788930384.231890 mapwrite 0x0c0000 thru 0x0ca0fa (0xa0fb bytes) 000101[1] 1788930384.325321 mapwrite 0x8d6000 thru 0x8dbabb (0x5abc bytes) 000102[0] 1788930384.451150 punch from 0x4c811e to 0x4d57fa (0xd6dd bytes) 000103[1] 1788930384.495814 trunc from 0x8dbabc to 0x3759f8 (0x5660c3 bytes) 000104[0] 1788930384.537362 trunc from 0x3759f9 to 0x1a3a84 (0x1d1f74 bytes) 000105[0] 1788930384.590946 mapwrite 0x87c000 thru 0x88a750 (0xe751 bytes) 000108[0] 1788930384.769133 mapread 0x675000 thru 0x6841da (0xf1db bytes) 000109[0] 1788930384.878956 punch from 0x15c4ff to 0x1653b1 (0x8eb3 bytes) 000110[1] 1788930384.945252 mapread 0x5ef000 thru 0x5f7f81 (0x8f82 bytes) 000115[1] 1788930385.064219 mapread 0x56c000 thru 0x57b3d2 (0xf3d3 bytes) 000117[0] 1788930385.128792 trunc from 0x88a751 to 0x6369a9 (0x253da7 bytes) 000118[1] 1788930385.259287 mapread 0x556000 thru 0x563d16 (0xdd17 bytes) 000121[1] 1788930385.330247 trunc from 0x6369aa to 0x05baa2 (0x5daf07 bytes) 000122[1] 1788930385.373128 write 0x363000 thru 0x367046 (0x4047 bytes) 000123[1] 1788930385.398249 mapread 0x246000 thru 0x246f49 (0xf4a bytes) 000126[1] 1788930385.404583 read 0x2e2000 thru 0x2e623a (0x423b bytes) 000128[1] 1788930385.487631 read 0x1d4000 thru 0x1e0e21 (0xce22 bytes) 000129[1] 1788930385.503005 mapwrite 0x97d000 thru 0x9821f7 (0x51f8 bytes) 000134[1] 1788930385.743296 mapread 0x7f0000 thru 0x7f0036 (0x37 bytes) 000135[0] 1788930385.744048 mapread 0x6fe000 thru 0x70275d (0x475e bytes) 000138[1] 1788930385.792103 mapwrite 0x892000 thru 0x8a1ba5 (0xfba6 bytes) 000141[1] 1788930385.990706 mapwrite 0x893000 thru 0x895d4d (0x2d4e bytes) 000142[1] 1788930386.101582 punch from 0x91009e to 0x91921d (0x9180 bytes) 000143[1] 1788930386.159925 trunc from 0x9821f8 to 0x4e0440 (0x4a1db7 bytes) 000147[0] 1788930386.237857 read 0x08e000 thru 0x092b2c (0x4b2d bytes) 000149[0] 1788930386.281600 trunc from 0x4e0441 to 0x0fc3f6 (0x3e404a bytes) 000150[0] 1788930386.330356 write 0x398000 thru 0x3a0425 (0x8426 bytes) 000151[0] 1788930386.364264 write 0x74f000 thru 0x755ab5 (0x6ab6 bytes) 000154[0] 1788930386.421242 read 0x5bb000 thru 0x5c8ee3 (0xdee4 bytes) 000155[0] 1788930386.432148 read 0x279000 thru 0x280ea2 (0x7ea3 bytes) 000156[0] 1788930386.441080 mapwrite 0x2c3000 thru 0x2c83dc (0x53dd bytes) 000157[0] 1788930386.543250 write 0x895000 thru 0x897df6 (0x2df7 bytes) 000161[0] 1788930386.584606 write 0x0b4000 thru 0x0b675a (0x275b bytes) 000162[0] 1788930386.596697 mapread 0x024000 thru 0x031b10 (0xdb11 bytes) 000163[1] 1788930386.764830 mapread 0x49d000 thru 0x4a8a7c (0xba7d bytes) 000165[1] 1788930386.987336 mapwrite 0x975000 thru 0x97ff63 (0xaf64 bytes) 000166[0] 1788930387.246280 mapwrite 0x29d000 thru 0x2a8493 (0xb494 bytes) 000167[1] 1788930387.363115 read 0x7c5000 thru 0x7c999c (0x499d bytes) 000169[0] 1788930387.396742 mapread 0x0bb000 thru 0x0c36f3 (0x86f4 bytes) 000170[0] 1788930387.425902 mapwrite 0x18e000 thru 0x19dfd5 (0xffd6 bytes) 000171[0] 1788930387.544216 mapwrite 0x20d000 thru 0x21a99c (0xd99d bytes) 000172[0] 1788930387.656017 mapwrite 0x6e5000 thru 0x6ed419 (0x841a bytes) 000174[1] 1788930387.730017 trunc from 0x97ff64 to 0x75f848 (0x22071b bytes) 000177[1] 1788930387.802107 read 0x134000 thru 0x13ff47 (0xbf48 bytes) 000180[1] 1788930387.862253 mapwrite 0x4ff000 thru 0x50b962 (0xc963 bytes) 000182[0] 1788930387.973091 read 0x431000 thru 0x43f50b (0xe50c bytes) 000183[0] 1788930388.004920 read 0x654000 thru 0x6616d7 (0xd6d8 bytes) 000187[0] 1788930388.016426 mapread 0x47d000 thru 0x48a6d0 (0xd6d1 bytes) 000190[0] 1788930388.054561 mapread 0x49e000 thru 0x4a08b2 (0x28b3 bytes) 000191[0] 1788930388.065168 read 0x0cf000 thru 0x0d6ab7 (0x7ab8 bytes) 000192[0] 1788930388.077834 read 0x58b000 thru 0x58e43a (0x343b bytes) 000193[1] 1788930388.086064 mapwrite 0x546000 thru 0x549758 (0x3759 bytes) 000194[1] 1788930388.182788 write 0x5f1000 thru 0x5ff3d0 (0xe3d1 bytes) 000195[0] 1788930388.193911 trunc from 0x75f849 to 0x38d0f9 (0x3d274f bytes) 000196[1] 1788930388.265818 trunc from 0x38d0fa to 0x348ac8 (0x44631 bytes) 000197[0] 1788930388.312677 write 0x89f000 thru 0x8a8863 (0x9864 bytes) 000198[1] 1788930388.357309 mapwrite 0x397000 thru 0x3a1367 (0xa368 bytes) 000199[0] 1788930388.491276 punch from 0x168e30 to 0x1709fe (0x7bcf bytes) 000200[0] 1788930388.546556 punch from 0x5cc469 to 0x5d5c8e (0x9826 bytes) 000201[1] 1788930388.577055 mapread 0x33d000 thru 0x348f2e (0xbf2f bytes) 000202[0] 1788930388.629774 punch from 0x1701c4 to 0x170395 (0x1d2 bytes) 000203[1] 1788930388.676764 mapread 0x748000 thru 0x75614e (0xe14f bytes) 000205[0] 1788930388.773413 write 0x9dc000 thru 0x9dea57 (0x2a58 bytes) 000209[0] 1788930388.889028 mapwrite 0x7e6000 thru 0x7e6101 (0x102 bytes) 000210[0] 1788930388.991238 read 0x085000 thru 0x086112 (0x1113 bytes) 000211[0] 1788930389.015644 punch from 0x995c15 to 0x9994b1 (0x389d bytes) 000213[1] 1788930389.055040 mapwrite 0x9a0000 thru 0x9a2ab0 (0x2ab1 bytes) 000214[0] 1788930389.178118 read 0x620000 thru 0x628791 (0x8792 bytes) 000215[0] 1788930389.208272 mapread 0x70f000 thru 0x71b33c (0xc33d bytes) 000216[0] 1788930389.227593 read 0x690000 thru 0x69bf48 (0xbf49 bytes) 000220[1] 1788930389.240040 mapread 0x2d7000 thru 0x2e4b3c (0xdb3d bytes) 000221[0] 1788930389.275658 write 0x322000 thru 0x32c2c9 (0xa2ca bytes) 000223[1] 1788930389.329761 write 0x725000 thru 0x7302c2 (0xb2c3 bytes) 000224[0] 1788930389.360666 mapread 0x7ac000 thru 0x7b022a (0x422b bytes) 000225[0] 1788930389.423236 mapwrite 0x433000 thru 0x440ff7 (0xdff8 bytes) 000226[1] 1788930389.557422 mapwrite 0x4fd000 thru 0x5034ce (0x64cf bytes) 000227[1] 1788930389.758154 mapread 0x46b000 thru 0x46bb6f (0xb70 bytes) 000229[1] 1788930389.793988 punch from 0x757298 to 0x75b75a (0x44c3 bytes) 000230[1] 1788930389.843177 write 0x6d1000 thru 0x6e089c (0xf89d bytes) 000231[1] 1788930389.888493 write 0x88e000 thru 0x8977e1 (0x97e2 bytes) 000232[0] 1788930389.907881 mapwrite 0x05a000 thru 0x05d16d (0x316e bytes) 000233[0] 1788930390.256578 read 0x38b000 thru 0x396458 (0xb459 bytes) 000234[0] 1788930390.294434 mapread 0x87b000 thru 0x87f278 (0x4279 bytes) 000235[0] 1788930390.347570 mapwrite 0x332000 thru 0x337385 (0x5386 bytes) 000236[0] 1788930390.529617 read 0x4ca000 thru 0x4cd556 (0x3557 bytes) 000237[0] 1788930390.560011 mapwrite 0x079000 thru 0x0861ce (0xd1cf bytes) 000239[1] 1788930390.759176 trunc from 0x9dea58 to 0x803227 (0x1db830 bytes) 000240[0] 1788930390.799027 mapwrite 0x355000 thru 0x356ce6 (0x1ce7 bytes) 000242[0] 1788930391.075930 trunc from 0x803228 to 0x8c551e (0xc22f7 bytes) 000243[0] 1788930391.123664 read 0x122000 thru 0x1309d3 (0xe9d4 bytes) 000250[1] 1788930391.191703 mapread 0x6f7000 thru 0x6f76ce (0x6cf bytes) 000251[0] 1788930391.223470 read 0x5d1000 thru 0x5d21c0 (0x11c1 bytes) 000252[0] 1788930391.239625 read 0x7e6000 thru 0x7e69e7 (0x9e8 bytes) 000253[1] 1788930391.246760 punch from 0x73f311 to 0x7443ba (0x50aa bytes) 000254[0] 1788930391.295639 trunc from 0x8c551f to 0x7b6935 (0x10ebe9 bytes) 000257[1] 1788930391.371118 write 0x8a3000 thru 0x8a5c5a (0x2c5b bytes) 000259[0] 1788930391.444050 read 0x3eb000 thru 0x3f2010 (0x7011 bytes) 000261[1] 1788930391.489202 write 0x032000 thru 0x03615d (0x415e bytes) 000267[0] 1788930391.672357 punch from 0x4ea7b9 to 0x4f9c0c (0xf454 bytes) 000268[1] 1788930391.755107 write 0x01c000 thru 0x01ea92 (0x2a93 bytes) 000271[0] 1788930391.808244 mapread 0x627000 thru 0x6297d3 (0x27d4 bytes) 000274[0] 1788930391.931132 write 0x8a6000 thru 0x8ab680 (0x5681 bytes) 000275[0] 1788930391.989299 trunc from 0x8ab681 to 0x33d391 (0x56e2ef bytes) 000282[0] 1788930392.216635 write 0x49a000 thru 0x4a13d4 (0x73d5 bytes) 000283[1] 1788930392.241076 punch from 0x0c1ba4 to 0x0d0fec (0xf449 bytes) 000284[1] 1788930392.291851 read 0x3bb000 thru 0x3c4ef3 (0x9ef4 bytes) 000288[0] 1788930392.361230 trunc from 0x4a13d5 to 0x4a3fe7 (0x2c13 bytes) 000289[1] 1788930392.412178 punch from 0x1e989c to 0x1f119b (0x7900 bytes) 000290[1] 1788930392.452207 write 0x8e4000 thru 0x8e4ee1 (0xee2 bytes) 000291[1] 1788930392.466329 read 0x46b000 thru 0x46fbcc (0x4bcd bytes) 000293[0] 1788930392.527510 punch from 0x0f3edc to 0x103d7f (0xfea4 bytes) 000295[1] 1788930392.552503 trunc from 0x8e4ee2 to 0x5e06f8 (0x3047e9 bytes) 000296[1] 1788930392.601148 write 0x49c000 thru 0x4a7d6f (0xbd70 bytes) 000299[0] 1788930392.644775 trunc from 0x5e06f9 to 0x061463 (0x57f295 bytes) 000300[0] 1788930392.743450 trunc from 0x061464 to 0x6238a1 (0x5c243e bytes) 000301[0] 1788930392.771229 trunc from 0x6238a2 to 0x7445d9 (0x120d38 bytes) 000303[0] 1788930392.795305 write 0x302000 thru 0x307e93 (0x5e94 bytes) 000304[0] 1788930392.831738 write 0x9bd000 thru 0x9cb40a (0xe40b bytes) 000306[0] 1788930392.864968 mapread 0x906000 thru 0x914afe (0xeaff bytes) 000308[1] 1788930392.950673 write 0x49f000 thru 0x4a89ef (0x99f0 bytes) 000309[1] 1788930393.018847 write 0x68b000 thru 0x690b2d (0x5b2e bytes) 000312[1] 1788930393.063485 write 0x1c9000 thru 0x1d53fc (0xc3fd bytes) 000313[1] 1788930393.074840 punch from 0x62a177 to 0x63886d (0xe6f7 bytes) 000314[1] 1788930393.095079 mapwrite 0x3bb000 thru 0x3bc323 (0x1324 bytes) 000315[0] 1788930393.193039 trunc from 0x9cb40b to 0x174502 (0x856f08 bytes) 000316[0] 1788930393.268204 mapwrite 0x327000 thru 0x329523 (0x2524 bytes) 000318[1] 1788930393.447033 write 0x171000 thru 0x171f00 (0xf01 bytes) 000319[1] 1788930393.541345 punch from 0x29a221 to 0x2a8b7a (0xe95a bytes) 000320[0] 1788930393.548301 read 0x283000 thru 0x28fd1e (0xcd1f bytes) 000323[0] 1788930393.681484 punch from 0x25de95 to 0x2681d2 (0xa33e bytes) 000324[1] 1788930393.754817 write 0x77e000 thru 0x7861b0 (0x81b1 bytes) 000325[1] 1788930393.809364 mapread 0x2e8000 thru 0x2eda71 (0x5a72 bytes) 000327[1] 1788930393.828216 read 0x586000 thru 0x5867b2 (0x7b3 bytes) 000328[0] 1788930393.848715 mapwrite 0x5c5000 thru 0x5c753b (0x253c bytes) 000329[0] 1788930394.031496 write 0x0ae000 thru 0x0b11b6 (0x31b7 bytes) 000330[1] 1788930394.041339 read 0x263000 thru 0x26f692 (0xc693 bytes) 000332[1] 1788930394.096043 write 0x26a000 thru 0x26eff3 (0x4ff4 bytes) 000333[0] 1788930394.131186 mapread 0x3d8000 thru 0x3e71e8 (0xf1e9 bytes) 000335[0] 1788930394.324093 mapwrite 0x056000 thru 0x060fe8 (0xafe9 bytes) 000337[0] 1788930394.531801 read 0x1d8000 thru 0x1e66c5 (0xe6c6 bytes) 000338[1] 1788930394.588004 read 0x70a000 thru 0x714873 (0xa874 bytes) 000340[0] 1788930394.628770 trunc from 0x7861b1 to 0x19130f (0x5f4ea1 bytes) 000341[1] 1788930394.699624 punch from 0x041227 to 0x04363b (0x2415 bytes) 000342[0] 1788930394.732705 read 0x068000 thru 0x075e48 (0xde49 bytes) 000343[1] 1788930394.813973 mapwrite 0x456000 thru 0x45b98b (0x598c bytes) 000345[0] 1788930395.007120 mapwrite 0x7e6000 thru 0x7f5c39 (0xfc3a bytes) 000346[1] 1788930395.378814 read 0x1fa000 thru 0x2072a0 (0xd2a1 bytes) 000347[1] 1788930395.441877 read 0x547000 thru 0x547a96 (0xa97 bytes) 000350[0] 1788930395.468668 mapwrite 0x04c000 thru 0x05ae8c (0xee8d bytes) 000351[0] 1788930395.747457 write 0x953000 thru 0x95c31f (0x9320 bytes) 000352[1] 1788930395.781164 write 0x9f0000 thru 0x9f7c53 (0x7c54 bytes) 000353[0] 1788930395.814264 mapwrite 0x42c000 thru 0x43029e (0x429f bytes) 000354[0] 1788930395.955364 trunc from 0x9f7c54 to 0x5281c6 (0x4cfa8d bytes) 000356[0] 1788930396.015111 trunc from 0x5281c7 to 0x6a2d8c (0x17abc6 bytes) 000357[0] 1788930396.076620 mapread 0x63d000 thru 0x6452e2 (0x82e3 bytes) 000358[1] 1788930396.282739 punch from 0x200c1c to 0x2039b9 (0x2d9e bytes) 000359[0] 1788930396.313966 mapread 0x183000 thru 0x1904b0 (0xd4b1 bytes) 000360[0] 1788930396.551290 write 0x6e9000 thru 0x6ed40c (0x440d bytes) 000361[0] 1788930396.598855 trunc from 0x6ed40d to 0x76ef8e (0x81b82 bytes) 000362[1] 1788930396.635308 mapwrite 0x345000 thru 0x35227d (0xd27e bytes) 000363[0] 1788930396.872126 read 0x518000 thru 0x51d3de (0x53df bytes) 000364[1] 1788930396.931614 mapread 0x607000 thru 0x60a9f7 (0x39f8 bytes) 000365[0] 1788930396.988287 write 0x53d000 thru 0x53fbd9 (0x2bda bytes) 000366[0] 1788930397.056999 mapwrite 0x8ba000 thru 0x8c493b (0xa93c bytes) 000367[1] 1788930397.230498 trunc from 0x8c493c to 0x67a12a (0x24a811 bytes) 000368[1] 1788930397.296849 write 0x05a000 thru 0x05aa10 (0xa11 bytes) 000369[1] 1788930397.357552 trunc from 0x67a12b to 0x2be4ea (0x3bbc40 bytes) 000370[0] 1788930397.432019 mapwrite 0x6ae000 thru 0x6bd711 (0xf712 bytes) 000372[1] 1788930397.692139 read 0x472000 thru 0x4810ac (0xf0ad bytes) 000374[1] 1788930397.745636 read 0x23a000 thru 0x23b597 (0x1598 bytes) 000378[1] 1788930397.755244 trunc from 0x6bd712 to 0x368d20 (0x3549f1 bytes) 000380[1] 1788930397.828944 read 0x289000 thru 0x290c69 (0x7c6a bytes) 000383[0] 1788930397.939452 read 0x1dc000 thru 0x1e6d11 (0xad12 bytes) 000384[0] 1788930398.043241 mapread 0x04b000 thru 0x04fb47 (0x4b48 bytes) 000386[1] 1788930398.044626 write 0x9d0000 thru 0x9de6f9 (0xe6fa bytes) 000387[0] 1788930398.070540 trunc from 0x9de6fa to 0x030b7d (0x9adb7c bytes) 000388[1] 1788930398.129242 trunc from 0x030b7e to 0x34c528 (0x31b9ab bytes) 000389[0] 1788930398.186894 mapwrite 0x9c5000 thru 0x9d403f (0xf040 bytes) 000390[1] 1788930398.480152 mapread 0x39d000 thru 0x3a43e4 (0x73e5 bytes) 000391[0] 1788930398.598146 trunc from 0x9d4040 to 0x4593cb (0x57ac74 bytes) 000393[1] 1788930398.703080 trunc from 0x4593cc to 0x594b3b (0x13b770 bytes) truncating to largest ever: 0x9deaf6 000394[1] 1788930398.872116 trunc from 0x594b3c to 0x9deaf5 (0x449fba bytes) 000395[0] 1788930398.940763 read 0x627000 thru 0x62890d (0x190e bytes) 000396[1] 1788930398.993628 write 0x726000 thru 0x726608 (0x609 bytes) 000397[0] 1788930399.027919 write 0x634000 thru 0x63f568 (0xb569 bytes) 000398[0] 1788930399.090359 mapwrite 0x4e2000 thru 0x4ea4e2 (0x84e3 bytes) 000399[1] 1788930399.308158 mapread 0x5ab000 thru 0x5b49e3 (0x99e4 bytes) 000400[1] 1788930399.424956 mapwrite 0x724000 thru 0x72a57c (0x657d bytes) 000401[0] 1788930399.536640 mapread 0x032000 thru 0x03e405 (0xc406 bytes) 000403[1] 1788930399.595855 mapwrite 0x715000 thru 0x71a1ea (0x51eb bytes) 000404[0] 1788930399.786884 mapwrite 0x758000 thru 0x762e7d (0xae7e bytes) 000405[1] 1788930399.988443 mapread 0x127000 thru 0x134a3e (0xda3f bytes) 000406[0] 1788930400.143235 write 0x68e000 thru 0x694ea8 (0x6ea9 bytes) 000407[1] 1788930400.190855 mapwrite 0x80e000 thru 0x8180b1 (0xa0b2 bytes) 000410[0] 1788930400.397056 write 0x3da000 thru 0x3e060c (0x660d bytes) 000411[1] 1788930400.507855 punch from 0x4f8df8 to 0x501564 (0x876d bytes) 000412[1] 1788930400.618155 mapread 0x7d2000 thru 0x7d8be9 (0x6bea bytes) 000414[1] 1788930400.761768 trunc from 0x9deaf6 to 0x8170a2 (0x1c7a53 bytes) 000416[1] 1788930400.830182 trunc from 0x8170a3 to 0x30851e (0x50eb84 bytes) 000418[0] 1788930400.856553 trunc from 0x30851f to 0x42ce9b (0x12497d bytes) 000419[1] 1788930400.933823 punch from 0x33dfc0 to 0x34d1ca (0xf20b bytes) 000420[0] 1788930400.971525 mapread 0x258000 thru 0x2630bd (0xb0be bytes) 000422[1] 1788930401.084810 mapwrite 0x7bb000 thru 0x7ca72d (0xf72e bytes) 000423[0] 1788930401.404700 punch from 0x225357 to 0x229141 (0x3deb bytes) 000425[1] 1788930401.459683 mapwrite 0x7ce000 thru 0x7dd3c7 (0xf3c8 bytes) 000426[0] 1788930401.673171 read 0x4b4000 thru 0x4ba469 (0x646a bytes) 000427[0] 1788930401.703477 mapread 0x428000 thru 0x437787 (0xf788 bytes) 000430[0] 1788930401.739345 mapread 0x0e0000 thru 0x0e3f7c (0x3f7d bytes) 000431[1] 1788930401.759484 mapread 0x66e000 thru 0x67a1a7 (0xc1a8 bytes) 000433[1] 1788930401.833847 read 0x0bf000 thru 0x0bff2b (0xf2c bytes) 000435[0] 1788930401.841597 mapwrite 0x336000 thru 0x344c7a (0xec7b bytes) 000437[1] 1788930401.965792 punch from 0x0fb4aa to 0x107369 (0xbec0 bytes) 000439[1] 1788930402.007734 write 0x534000 thru 0x54146c (0xd46d bytes) 000441[1] 1788930402.055069 read 0x264000 thru 0x2665fb (0x25fc bytes) 000442[0] 1788930402.065639 write 0x1f1000 thru 0x1f47ba (0x37bb bytes) 000443[1] 1788930402.123329 write 0x5c3000 thru 0x5ce08a (0xb08b bytes) 000444[0] 1788930402.151807 mapwrite 0x2a3000 thru 0x2ac616 (0x9617 bytes) 000445[1] 1788930402.306034 mapwrite 0x24a000 thru 0x25274f (0x8750 bytes) 000447[1] 1788930402.433130 trunc from 0x7dd3c8 to 0x5addde (0x22f5e9 bytes) 000450[0] 1788930402.481625 mapread 0x2a2000 thru 0x2b1248 (0xf249 bytes) 000452[0] 1788930402.534498 mapwrite 0x5fe000 thru 0x608161 (0xa162 bytes) 000453[0] 1788930402.699267 write 0x3c0000 thru 0x3cbe2c (0xbe2d bytes) 000454[1] 1788930402.706634 read 0x39c000 thru 0x39c7e4 (0x7e5 bytes) 000455[0] 1788930402.749281 punch from 0x0f3669 to 0x0f68ca (0x3262 bytes) 000456[0] 1788930402.807118 trunc from 0x608162 to 0x04f20c (0x5b8f55 bytes) 000457[0] 1788930402.845414 mapread 0x007000 thru 0x00cf34 (0x5f35 bytes) 000459[1] 1788930402.888356 trunc from 0x04f20d to 0x29d2b1 (0x24e0a5 bytes) 000460[0] 1788930402.929054 mapwrite 0x815000 thru 0x8231cd (0xe1ce bytes) 000462[1] 1788930403.078849 trunc from 0x8231ce to 0x2d3770 (0x54fa5d bytes) 000463[0] 1788930403.116555 mapread 0x113000 thru 0x1130ca (0xcb bytes) 000467[1] 1788930403.150645 punch from 0x1ae9f7 to 0x1bb92f (0xcf39 bytes) 000469[1] 1788930403.189246 mapread 0x213000 thru 0x2173de (0x43df bytes) 000470[0] 1788930403.252289 punch from 0x2c6052 to 0x2d2d29 (0xccd8 bytes) 000472[0] 1788930403.299554 write 0x19a000 thru 0x1a8b35 (0xeb36 bytes) 000473[0] 1788930403.321426 mapread 0x00b000 thru 0x016982 (0xb983 bytes) 000474[1] 1788930403.351266 mapread 0x051000 thru 0x055871 (0x4872 bytes) 000477[1] 1788930403.421192 write 0x3dd000 thru 0x3eb743 (0xe744 bytes) 000478[0] 1788930403.455096 write 0x2de000 thru 0x2e7179 (0x917a bytes) 000479[0] 1788930403.504729 punch from 0x231516 to 0x23fc60 (0xe74b bytes) 000482[1] 1788930403.559126 punch from 0x0f8de5 to 0x1074e7 (0xe703 bytes) 000485[0] 1788930403.615021 trunc from 0x3eb744 to 0x7fecf6 (0x4135b3 bytes) 000487[0] 1788930403.690265 mapwrite 0x9d9000 thru 0x9e3f79 (0xaf7a bytes) 000489[1] 1788930403.815316 trunc from 0x9e3f7a to 0x31779d (0x6cc7dc bytes) 000491[0] 1788930403.881082 mapwrite 0x998000 thru 0x998e4a (0xe4b bytes) 000492[1] 1788930404.049875 read 0x4a2000 thru 0x4ab11b (0x911c bytes) 000493[1] 1788930404.095199 trunc from 0x998e4b to 0x0b49bc (0x8e448e bytes) 000494[1] 1788930404.148013 trunc from 0x0b49bd to 0x80f9f1 (0x75b035 bytes) 000495[1] 1788930404.177788 punch from 0x078e38 to 0x0842a7 (0xb470 bytes) 000496[1] 1788930404.239105 read 0x1fb000 thru 0x204eee (0x9eef bytes) 000498[1] 1788930404.330911 mapwrite 0x651000 thru 0x65f0bc (0xe0bd bytes) 000500[0] 1788930404.522460 mapwrite 0x107000 thru 0x10cd27 (0x5d28 bytes) All operations completed A-OK! PASS 16a (62s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 01:06:51 (1788930411) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.41363 s, 20.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.323349 s, 25.9 MB/s Chance of close/open is 1 in 50 Seed set to 415 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x4a40f5 000002[1] 1788930416.473720 trunc from 00000000 to 0x4a40f4 (0x4a40f5 bytes) 000004[0] 1788930416.574727 write 0x5bd88c thru 0x5be03c (0x7b1 bytes) 000005[0] 1788930416.633860 mapwrite 0x4c55e8 thru 0x4c7211 (0x1c2a bytes) 000008[1] 1788930416.937590 mapwrite 0x746014 thru 0x746499 (0x486 bytes) 000010[0] 1788930417.115659 trunc from 0x74649a to 0x4604f6 (0x2e5fa3 bytes) 000013[1] 1788930417.241834 mapread 0x38b82a thru 0x38c627 (0xdfe bytes) 000017[1] 1788930417.332741 write 0x17454c thru 0x175e83 (0x1938 bytes) 000018[0] 1788930417.410748 punch from 0x3b34be to 0x3b5267 (0x1daa bytes) 000019[1] 1788930417.519744 punch from 0x1a3f0a to 0x1a415c (0x253 bytes) 000020[1] 1788930417.559602 read 0x3ff24b thru 0x3ffaf1 (0x8a7 bytes) 000021[1] 1788930417.579254 mapread 0x00b224 thru 0x00b237 (0x14 bytes) 000023[0] 1788930417.625702 read 0x226bf3 thru 0x227beb (0xff9 bytes) truncating to largest ever: 0x9e0151 000025[1] 1788930417.686298 trunc from 0x4604f7 to 0x9e0150 (0x57fc5a bytes) 000026[1] 1788930417.774859 read 0x636bc9 thru 0x637d8c (0x11c4 bytes) 000035[1] 1788930417.881030 write 0x5079c3 thru 0x50961c (0x1c5a bytes) 000040[1] 1788930417.897857 trunc from 0x9e0151 to 0x31d1cd (0x6c2f83 bytes) 000041[0] 1788930417.922118 mapwrite 0x05d559 thru 0x05f455 (0x1efd bytes) 000043[1] 1788930418.058034 write 0x098fbe thru 0x09a338 (0x137b bytes) 000045[1] 1788930418.126910 mapread 0x0b55e3 thru 0x0b56fe (0x11c bytes) 000047[0] 1788930418.137041 trunc from 0x31d1ce to 0x67337c (0x3561af bytes) 000048[1] 1788930418.273360 mapwrite 0x7c60a4 thru 0x7c65ef (0x54c bytes) 000049[1] 1788930418.404178 punch from 0x30d0bd to 0x30e21b (0x115f bytes) 000050[0] 1788930418.413140 mapwrite 0x13e278 thru 0x13e618 (0x3a1 bytes) 000051[1] 1788930418.576721 mapread 0x5b730d thru 0x5b902a (0x1d1e bytes) 000054[1] 1788930418.694292 punch from 0x0d5c61 to 0x0d6eb7 (0x1257 bytes) 000056[1] 1788930418.790282 read 0x16c7ff thru 0x16d77b (0xf7d bytes) 000059[1] 1788930418.840070 mapread 0x032b2f thru 0x032fe5 (0x4b7 bytes) 000063[1] 1788930418.986630 mapread 0x5c2721 thru 0x5c430d (0x1bed bytes) 000065[0] 1788930419.093608 trunc from 0x7c65f0 to 0x1218bc (0x6a4d33 bytes) 000067[1] 1788930419.260764 mapread 0x0b9997 thru 0x0bae75 (0x14df bytes) 000068[1] 1788930419.322304 read 0x0204e1 thru 0x021df8 (0x1918 bytes) 000070[0] 1788930419.369085 mapwrite 0x9d0c5a thru 0x9d2353 (0x16fa bytes) 000073[1] 1788930419.530284 trunc from 0x9d2354 to 0x4d6a4e (0x4fb905 bytes) 000076[0] 1788930419.628730 punch from 0x0d5b0e to 0x0d75f4 (0x1ae7 bytes) 000077[0] 1788930419.668980 read 0x385d1d thru 0x386958 (0xc3c bytes) 000078[0] 1788930419.703522 read 0x10a972 thru 0x10a98a (0x19 bytes) 000079[0] 1788930419.837899 punch from 0x233465 to 0x234ef8 (0x1a94 bytes) 000082[0] 1788930419.946469 mapread 0x222bd3 thru 0x224b25 (0x1f53 bytes) 000086[1] 1788930420.090823 read 0x12b3a9 thru 0x12bbc9 (0x821 bytes) 000087[0] 1788930420.129100 write 0x934f30 thru 0x9351c4 (0x295 bytes) 000088[0] 1788930420.197787 mapread 0x731c04 thru 0x732b4b (0xf48 bytes) 000091[0] 1788930420.399416 write 0x0f0c90 thru 0x0f1fe1 (0x1352 bytes) 000095[0] 1788930420.462993 read 0x32a807 thru 0x32ac8a (0x484 bytes) 000096[1] 1788930420.472115 mapwrite 0x7560e6 thru 0x757717 (0x1632 bytes) 000097[1] 1788930420.728588 mapread 0x44d640 thru 0x44de06 (0x7c7 bytes) 000101[1] 1788930420.873427 mapread 0x420787 thru 0x421fea (0x1864 bytes) 000104[0] 1788930420.993353 read 0x2522e4 thru 0x252380 (0x9d bytes) 000108[1] 1788930421.108326 punch from 0x237d19 to 0x238715 (0x9fd bytes) 000109[0] 1788930421.149193 write 0x069b43 thru 0x06b2c3 (0x1781 bytes) 000114[1] 1788930421.328870 write 0x8a720d thru 0x8a8592 (0x1386 bytes) 000116[0] 1788930421.435022 mapwrite 0x600b9a thru 0x602136 (0x159d bytes) 000121[0] 1788930421.828579 read 0x908c97 thru 0x908ce8 (0x52 bytes) 000122[0] 1788930421.903894 punch from 0x22aa10 to 0x22b5fd (0xbee bytes) 000123[0] 1788930421.952973 write 0x2ab0ba thru 0x2ab814 (0x75b bytes) 000124[1] 1788930422.015952 write 0x6c9084 thru 0x6cae70 (0x1ded bytes) 000125[0] 1788930422.108860 mapread 0x671825 thru 0x672aae (0x128a bytes) 000126[1] 1788930422.162194 punch from 0x7b92f5 to 0x7ba626 (0x1332 bytes) 000128[0] 1788930422.278631 punch from 0x0992c3 to 0x09b22f (0x1f6d bytes) 000130[0] 1788930422.353023 punch from 0x4d7636 to 0x4d8170 (0xb3b bytes) 000131[1] 1788930422.372026 trunc from 0x9351c5 to 0x42e7c7 (0x5069fd bytes) 000133[0] 1788930422.438080 trunc from 0x42e7c8 to 0x476d87 (0x485c0 bytes) 000134[1] 1788930422.482743 mapwrite 0x1093fc thru 0x10ab9d (0x17a2 bytes) 000135[1] 1788930422.621727 read 0x3b895e thru 0x3ba835 (0x1ed8 bytes) 000136[0] 1788930422.667779 write 0x016bda thru 0x018024 (0x144b bytes) 000137[0] 1788930422.790231 write 0x8c3824 thru 0x8c3cea (0x4c7 bytes) 000138[1] 1788930422.811980 read 0x43bed4 thru 0x43c9e7 (0xb14 bytes) 000140[0] 1788930422.892977 punch from 0x8529c3 to 0x8544ba (0x1af8 bytes) 000144[1] 1788930422.967075 mapwrite 0x44f757 thru 0x45083d (0x10e7 bytes) 000147[0] 1788930423.186464 read 0x77115f thru 0x771615 (0x4b7 bytes) 000149[0] 1788930423.251025 trunc from 0x8c3ceb to 0x4cdf67 (0x3f5d83 bytes) 000150[0] 1788930423.343219 trunc from 0x4cdf68 to 0x7a8b41 (0x2dabda bytes) 000151[1] 1788930423.380938 read 0x2f24f2 thru 0x2f3350 (0xe5f bytes) 000152[0] 1788930423.414693 read 0x1d9d74 thru 0x1da2f1 (0x57e bytes) 000155[0] 1788930423.514782 punch from 0x62a30b to 0x62b272 (0xf68 bytes) 000156[0] 1788930423.551740 punch from 0x5de826 to 0x5de9b4 (0x18f bytes) 000157[1] 1788930423.574353 mapwrite 0x652800 thru 0x65381d (0x101e bytes) 000163[1] 1788930423.735166 mapwrite 0x358c55 thru 0x35ab31 (0x1edd bytes) 000164[1] 1788930423.831941 read 0x33cd7d thru 0x33e4f9 (0x177d bytes) 000165[1] 1788930423.900488 punch from 0x773963 to 0x774c49 (0x12e7 bytes) 000171[1] 1788930424.073880 trunc from 0x7a8b42 to 0x236960 (0x5721e1 bytes) 000177[1] 1788930424.201112 trunc from 0x236961 to 0x57ef5f (0x3485ff bytes) 000179[1] 1788930424.310769 write 0x5953f3 thru 0x5958f4 (0x502 bytes) 000180[1] 1788930424.326103 mapread 0x576f41 thru 0x5789bc (0x1a7c bytes) 000181[1] 1788930424.365991 read 0x01dbcd thru 0x01ec69 (0x109d bytes) 000182[1] 1788930424.405805 write 0x32030c thru 0x321930 (0x1625 bytes) 000183[1] 1788930424.444857 read 0x3187d0 thru 0x318bb2 (0x3e3 bytes) 000184[0] 1788930424.455013 mapwrite 0x9b54be thru 0x9b72df (0x1e22 bytes) 000185[1] 1788930424.661108 write 0x3a8244 thru 0x3a86eb (0x4a8 bytes) 000186[0] 1788930424.731554 write 0x70df48 thru 0x70f9d8 (0x1a91 bytes) 000188[1] 1788930424.808242 write 0x2534cd thru 0x253918 (0x44c bytes) 000189[0] 1788930424.885618 write 0x8f25d4 thru 0x8f43f7 (0x1e24 bytes) 000190[0] 1788930424.981678 mapread 0x1c2457 thru 0x1c43cc (0x1f76 bytes) 000191[0] 1788930425.100975 mapread 0x09e330 thru 0x0a015c (0x1e2d bytes) 000193[1] 1788930425.194699 trunc from 0x9b72e0 to 0x4884d5 (0x52ee0a bytes) 000194[0] 1788930425.385690 trunc from 0x4884d6 to 0x47bbdc (0xc8f9 bytes) 000195[1] 1788930425.452256 punch from 0x0c6f03 to 0x0c7e02 (0xf00 bytes) 000196[0] 1788930425.486041 mapread 0x3dad5e thru 0x3dafa4 (0x247 bytes) 000197[1] 1788930425.540871 mapread 0x257b5c thru 0x25975f (0x1c04 bytes) 000199[0] 1788930425.645591 punch from 0x2a53f7 to 0x2a6add (0x16e7 bytes) 000200[0] 1788930425.669977 mapwrite 0x4a2faa thru 0x4a4414 (0x146b bytes) 000201[1] 1788930425.984546 mapwrite 0x7fd936 thru 0x7fe709 (0xdd4 bytes) 000202[1] 1788930426.196019 write 0x2542e1 thru 0x25462c (0x34c bytes) 000204[0] 1788930426.278505 write 0x84a72f thru 0x84c4de (0x1db0 bytes) 000205[1] 1788930426.317157 write 0x9e2b2f thru 0x9e3f05 (0x13d7 bytes) 000207[0] 1788930426.413080 trunc from 0x9e3f06 to 0x532479 (0x4b1a8c bytes) 000209[1] 1788930426.492657 mapwrite 0x5b64c9 thru 0x5b7ac7 (0x15ff bytes) 000210[0] 1788930426.632861 mapwrite 0x1cdbdf thru 0x1cdcc1 (0xe3 bytes) 000211[1] 1788930426.726748 read 0x4adfab thru 0x4af751 (0x17a7 bytes) 000212[1] 1788930426.763731 trunc from 0x5b7ac8 to 0x6cb010 (0x113549 bytes) 000213[1] 1788930426.819456 read 0x58fe8c thru 0x5918b4 (0x1a29 bytes) 000214[0] 1788930426.828112 write 0x9227b5 thru 0x9235e1 (0xe2d bytes) 000215[0] 1788930426.859987 punch from 0x4e8469 to 0x4e9e6f (0x1a07 bytes) 000216[1] 1788930426.898050 mapwrite 0x8afb03 thru 0x8aff8f (0x48d bytes) 000218[1] 1788930427.037075 mapwrite 0x95b028 thru 0x95c069 (0x1042 bytes) 000219[0] 1788930427.183883 write 0x50af17 thru 0x50ba32 (0xb1c bytes) 000220[1] 1788930427.235610 mapwrite 0x3ec343 thru 0x3ec478 (0x136 bytes) 000222[0] 1788930427.369923 mapread 0x04cf81 thru 0x04d2c7 (0x347 bytes) 000224[1] 1788930427.457512 punch from 0x19a18b to 0x19be5e (0x1cd4 bytes) 000225[0] 1788930427.501874 trunc from 0x95c06a to 0x371c54 (0x5ea415 bytes) 000228[1] 1788930427.640806 mapread 0x34f91c thru 0x34fd91 (0x476 bytes) 000229[1] 1788930427.680830 trunc from 0x371c55 to 0x8ac2f9 (0x53a6a5 bytes) 000230[0] 1788930427.763215 mapwrite 0x61d0fc thru 0x61dd4b (0xc50 bytes) 000231[0] 1788930427.886461 mapread 0x6bdd16 thru 0x6beeb3 (0x119e bytes) 000232[1] 1788930427.938798 mapwrite 0x364cd1 thru 0x365dff (0x112f bytes) 000233[0] 1788930428.089793 mapwrite 0x5725d3 thru 0x572918 (0x346 bytes) 000234[0] 1788930428.232032 write 0x23efd9 thru 0x23f4cd (0x4f5 bytes) 000235[1] 1788930428.266942 mapread 0x155877 thru 0x15724c (0x19d6 bytes) 000236[0] 1788930428.350589 trunc from 0x8ac2fa to 0x0988ab (0x813a4e bytes) 000238[0] 1788930428.423477 trunc from 0x0988ac to 0x1e3322 (0x14aa77 bytes) 000239[1] 1788930428.468115 read 0x0bc05c thru 0x0bceac (0xe51 bytes) 000241[0] 1788930428.561766 trunc from 0x1e3323 to 0x58221e (0x39eefc bytes) 000242[0] 1788930428.609565 mapwrite 0x49aae4 thru 0x49ae84 (0x3a1 bytes) 000244[0] 1788930428.704197 write 0x304902 thru 0x3060f1 (0x17f0 bytes) 000245[1] 1788930428.760075 read 0x2a5820 thru 0x2a5e80 (0x661 bytes) 000248[1] 1788930428.817769 write 0x64feae thru 0x65111e (0x1271 bytes) 000249[0] 1788930428.862568 trunc from 0x65111f to 0x3e972b (0x2679f3 bytes) 000250[1] 1788930428.932074 mapwrite 0x680cad thru 0x680eaf (0x203 bytes) 000252[0] 1788930429.129109 read 0x2aab74 thru 0x2aab7f (0xc bytes) 000253[1] 1788930429.201049 write 0x7a0dd3 thru 0x7a17f6 (0xa24 bytes) 000254[1] 1788930429.249753 mapwrite 0x2037c3 thru 0x203efe (0x73c bytes) 000255[1] 1788930429.396095 trunc from 0x7a17f7 to 0x921fa4 (0x1807ae bytes) 000256[0] 1788930429.481300 punch from 0x2330bf to 0x233397 (0x2d9 bytes) 000258[1] 1788930429.586004 mapwrite 0x948df7 thru 0x94ada6 (0x1fb0 bytes) 000259[1] 1788930429.822769 read 0x4cb5f7 thru 0x4cd3d1 (0x1ddb bytes) 000260[0] 1788930429.832963 mapwrite 0x354b78 thru 0x3562e9 (0x1772 bytes) 000261[1] 1788930430.063464 write 0x496b5e thru 0x498267 (0x170a bytes) 000262[1] 1788930430.125535 mapwrite 0x405b37 thru 0x407235 (0x16ff bytes) 000267[0] 1788930430.442033 mapwrite 0x2d1e6f thru 0x2d2bcc (0xd5e bytes) 000268[1] 1788930430.668263 read 0x82dc75 thru 0x82f20d (0x1599 bytes) 000270[1] 1788930430.757676 mapread 0x7cda41 thru 0x7cec70 (0x1230 bytes) 000271[1] 1788930430.774957 mapwrite 0x8e9f54 thru 0x8eafe8 (0x1095 bytes) 000272[0] 1788930431.028255 mapread 0x5924f8 thru 0x59438a (0x1e93 bytes) 000275[1] 1788930431.163863 write 0x8bfca2 thru 0x8c1af7 (0x1e56 bytes) 000276[1] 1788930431.175499 mapwrite 0x258a1e thru 0x259e18 (0x13fb bytes) 000277[1] 1788930431.274549 mapread 0x0a8935 thru 0x0aa14f (0x181b bytes) 000279[0] 1788930431.374573 write 0x334e02 thru 0x33548b (0x68a bytes) 000281[0] 1788930431.462862 write 0x0ed977 thru 0x0edcc2 (0x34c bytes) 000283[1] 1788930431.494541 trunc from 0x94ada7 to 0x3bfdb9 (0x58afed bytes) 000285[0] 1788930431.594307 write 0x0dd662 thru 0x0de1a0 (0xb3f bytes) 000287[1] 1788930431.669340 write 0x347e65 thru 0x348f80 (0x111c bytes) 000288[0] 1788930431.701930 mapwrite 0x36f44d thru 0x37131c (0x1ed0 bytes) 000289[1] 1788930431.818497 punch from 0x30dc46 to 0x30fbb7 (0x1f72 bytes) 000291[0] 1788930431.837710 mapread 0x191891 thru 0x191b03 (0x273 bytes) 000292[1] 1788930431.884036 mapwrite 0x618ded thru 0x61916f (0x383 bytes) 000294[1] 1788930432.042191 write 0x638aec thru 0x63a9c8 (0x1edd bytes) 000295[0] 1788930432.054146 mapwrite 0x2dae2f thru 0x2db63c (0x80e bytes) 000302[0] 1788930432.317923 read 0x032129 thru 0x032e07 (0xcdf bytes) 000303[1] 1788930432.342667 read 0x4313ec thru 0x433034 (0x1c49 bytes) 000305[0] 1788930432.389526 read 0x420b52 thru 0x421853 (0xd02 bytes) 000306[1] 1788930432.496242 punch from 0x114cb4 to 0x115c91 (0xfde bytes) 000307[1] 1788930432.605359 mapread 0x1217f9 thru 0x122a75 (0x127d bytes) 000308[0] 1788930432.648492 read 0x0438f1 thru 0x044395 (0xaa5 bytes) 000310[1] 1788930432.674769 mapwrite 0x4f0115 thru 0x4f1757 (0x1643 bytes) 000313[0] 1788930432.940628 mapread 0x1aecb0 thru 0x1afb6f (0xec0 bytes) 000315[0] 1788930433.042048 trunc from 0x63a9c9 to 0x23f840 (0x3fb188 bytes) 000316[0] 1788930433.140548 mapread 0x18a011 thru 0x18a50f (0x4ff bytes) 000317[1] 1788930433.171607 read 0x0ebea5 thru 0x0ec0f9 (0x255 bytes) 000319[1] 1788930433.237911 mapread 0x100f7e thru 0x102ef1 (0x1f74 bytes) 000320[1] 1788930433.276497 trunc from 0x23f841 to 0x2cb633 (0x8bdf3 bytes) 000321[0] 1788930433.349430 write 0x676988 thru 0x6772a5 (0x91e bytes) 000322[1] 1788930433.399965 read 0x224f88 thru 0x225d5a (0xdd3 bytes) 000327[1] 1788930433.636752 mapread 0x295816 thru 0x2969bd (0x11a8 bytes) 000328[1] 1788930433.664505 mapread 0x025eb2 thru 0x027575 (0x16c4 bytes) 000331[1] 1788930433.727528 mapwrite 0x52aece thru 0x52ce2c (0x1f5f bytes) 000332[0] 1788930433.846276 mapwrite 0x749dbd thru 0x749e76 (0xba bytes) 000333[1] 1788930433.980702 read 0x5bd7c6 thru 0x5bf722 (0x1f5d bytes) 000336[1] 1788930434.042477 punch from 0x4cc694 to 0x4cce29 (0x796 bytes) 000337[1] 1788930434.078745 mapread 0x6154fe thru 0x6157f1 (0x2f4 bytes) 000339[1] 1788930434.124642 mapwrite 0x8989ef thru 0x89a112 (0x1724 bytes) 000340[0] 1788930434.282994 read 0x455270 thru 0x45601e (0xdaf bytes) 000341[1] 1788930434.348103 mapwrite 0x5c71e3 thru 0x5c8828 (0x1646 bytes) 000343[1] 1788930434.526131 trunc from 0x89a113 to 0x1e34ee (0x6b6c24 bytes) 000344[1] 1788930434.581939 mapwrite 0x66f972 thru 0x670e8c (0x151b bytes) 000345[1] 1788930434.693558 trunc from 0x670e8d to 0x403db2 (0x26d0da bytes) 000346[0] 1788930434.738663 punch from 0x34b514 to 0x34bf0e (0x9fb bytes) 000347[1] 1788930434.804783 trunc from 0x403db3 to 0x3ffcdd (0x40d5 bytes) 000350[1] 1788930434.898932 mapwrite 0x020d6b thru 0x0212bd (0x553 bytes) 000352[0] 1788930435.074921 trunc from 0x3ffcde to 0x8dcbeb (0x4dcf0e bytes) 000353[0] 1788930435.157772 punch from 0x17fea0 to 0x181ddd (0x1f3e bytes) 000354[1] 1788930435.204358 mapread 0x5957fb thru 0x597405 (0x1c0b bytes) 000355[0] 1788930435.279634 mapwrite 0x9e2ea2 thru 0x9e46c2 (0x1821 bytes) 000357[1] 1788930435.451967 mapwrite 0x496aca thru 0x497202 (0x739 bytes) 000359[0] 1788930435.594742 mapwrite 0x0a7949 thru 0x0a7adc (0x194 bytes) 000361[0] 1788930435.710812 read 0x36329c thru 0x364753 (0x14b8 bytes) 000362[0] 1788930435.739970 write 0x53a108 thru 0x53ae1a (0xd13 bytes) 000363[1] 1788930435.762912 write 0x04c85c thru 0x04d6fa (0xe9f bytes) 000364[0] 1788930435.817873 write 0x4447ea thru 0x44563e (0xe55 bytes) 000365[1] 1788930435.844817 mapread 0x516612 thru 0x517784 (0x1173 bytes) 000367[1] 1788930435.892744 punch from 0x1999fa to 0x19ae16 (0x141d bytes) 000368[1] 1788930435.898794 mapread 0x0e483c thru 0x0e4def (0x5b4 bytes) 000370[0] 1788930435.927268 mapread 0x1ea4c9 thru 0x1eb8ab (0x13e3 bytes) 000372[0] 1788930435.968186 mapread 0x828cb3 thru 0x829a9c (0xdea bytes) 000374[1] 1788930435.989446 read 0x6b62d5 thru 0x6b7625 (0x1351 bytes) 000375[0] 1788930435.997266 punch from 0x165f31 to 0x166f3f (0x100f bytes) 000376[1] 1788930436.033499 write 0x1c5696 thru 0x1c640e (0xd79 bytes) 000378[1] 1788930436.080411 read 0x00e541 thru 0x00ff71 (0x1a31 bytes) 000379[0] 1788930436.108976 mapwrite 0x51ee12 thru 0x520bcc (0x1dbb bytes) 000381[1] 1788930436.260790 trunc from 0x9e46c3 to 0x27f538 (0x76518a bytes) 000382[1] 1788930436.315634 read 0x13659d thru 0x137132 (0xb96 bytes) 000384[1] 1788930436.429665 read 0x176a68 thru 0x177412 (0x9ab bytes) 000385[0] 1788930436.459755 mapread 0x1000c8 thru 0x1012eb (0x1224 bytes) 000386[0] 1788930436.513950 read 0x0b8500 thru 0x0b88bd (0x3be bytes) 000387[0] 1788930436.550713 mapread 0x170b0b thru 0x1714a4 (0x99a bytes) 000389[1] 1788930436.608940 trunc from 0x27f539 to 0x74b109 (0x4cbbd1 bytes) 000394[0] 1788930436.656156 trunc from 0x74b10a to 0x10d57a (0x63db8f bytes) 000395[1] 1788930436.728816 mapwrite 0x0bc306 thru 0x0bd32c (0x1027 bytes) 000397[0] 1788930436.848337 read 0x0db842 thru 0x0dd311 (0x1ad0 bytes) 000398[0] 1788930436.898572 punch from 0x0d2e89 to 0x0d3e64 (0xfdc bytes) 000399[0] 1788930436.942860 trunc from 0x10d57b to 0x730a0d (0x623493 bytes) 000400[1] 1788930436.970070 punch from 0x67e896 to 0x6800a5 (0x1810 bytes) 000401[1] 1788930437.015985 trunc from 0x730a0e to 0x0d3d7b (0x65cc92 bytes) 000402[0] 1788930437.059199 punch from 0x06b636 to 0x06d3f3 (0x1dbe bytes) 000403[1] 1788930437.089675 read 0x0a65f1 thru 0x0a74a7 (0xeb7 bytes) 000404[0] 1788930437.146160 punch from 0x00bbb6 to 0x00c4aa (0x8f5 bytes) 000405[0] 1788930437.182464 punch from 0x09bfb9 to 0x09c0cb (0x113 bytes) 000406[1] 1788930437.207188 mapwrite 0x282f9b thru 0x284dda (0x1e40 bytes) 000407[0] 1788930437.368271 mapread 0x1d0c10 thru 0x1d1369 (0x75a bytes) 000409[1] 1788930437.473219 mapread 0x10fa2e thru 0x110e66 (0x1439 bytes) 000410[1] 1788930437.519330 trunc from 0x284ddb to 0x9275e4 (0x6a280a bytes) 000412[0] 1788930437.599033 write 0x2cdac7 thru 0x2ce9e9 (0xf23 bytes) 000414[0] 1788930437.684973 mapwrite 0x396abb thru 0x3971bd (0x703 bytes) 000416[1] 1788930437.869566 punch from 0x05b60a to 0x05c6cd (0x10c4 bytes) 000418[1] 1788930437.941837 punch from 0x8e03e6 to 0x8e049f (0xba bytes) 000419[1] 1788930437.946682 mapwrite 0x868220 thru 0x8689df (0x7c0 bytes) 000421[1] 1788930438.112510 punch from 0x49e630 to 0x49f28f (0xc60 bytes) 000422[0] 1788930438.119030 mapwrite 0x241e63 thru 0x242b4a (0xce8 bytes) 000423[0] 1788930438.254289 mapread 0x158f42 thru 0x15adc8 (0x1e87 bytes) 000424[0] 1788930438.327929 mapread 0x539a43 thru 0x53a9e7 (0xfa5 bytes) 000427[1] 1788930438.700583 read 0x671169 thru 0x6713f5 (0x28d bytes) 000428[1] 1788930438.706913 mapwrite 0x11ef14 thru 0x120420 (0x150d bytes) 000431[0] 1788930438.868984 write 0x27e06e thru 0x27e278 (0x20b bytes) 000432[1] 1788930438.903014 read 0x494629 thru 0x4961bb (0x1b93 bytes) 000434[0] 1788930438.985581 write 0x02f42b thru 0x031249 (0x1e1f bytes) 000435[0] 1788930439.028772 trunc from 0x9275e5 to 0x7e506b (0x142579 bytes) 000436[0] 1788930439.117626 write 0x68481e thru 0x685039 (0x81c bytes) 000437[0] 1788930439.162896 trunc from 0x7e506c to 0x2320b4 (0x5b2fb7 bytes) 000438[1] 1788930439.190589 write 0x3b881c thru 0x3b964c (0xe31 bytes) 000439[1] 1788930439.235315 read 0x0dea2b thru 0x0deedb (0x4b1 bytes) 000440[0] 1788930439.261835 mapwrite 0x296fbf thru 0x297376 (0x3b8 bytes) 000442[1] 1788930439.381188 trunc from 0x3b964d to 0x41b2b3 (0x61c67 bytes) 000443[1] 1788930439.444671 trunc from 0x41b2b4 to 0x34b12a (0xd0189 bytes) 000444[0] 1788930439.464713 mapwrite 0x292773 thru 0x29456d (0x1dfb bytes) 000448[0] 1788930439.610283 trunc from 0x34b12b to 0x73e4a1 (0x3f3377 bytes) 000449[1] 1788930439.647313 write 0x4175c9 thru 0x417ea9 (0x8e1 bytes) 000450[1] 1788930439.679295 mapread 0x49d730 thru 0x49e374 (0xc45 bytes) 000454[0] 1788930439.786697 read 0x72b3d7 thru 0x72c6ca (0x12f4 bytes) 000455[0] 1788930439.820039 punch from 0x6f074a to 0x6f0d17 (0x5ce bytes) 000459[0] 1788930439.917684 mapwrite 0x4a678d thru 0x4a68c3 (0x137 bytes) 000460[1] 1788930439.988007 punch from 0x333339 to 0x334184 (0xe4c bytes) 000461[1] 1788930440.042716 mapwrite 0x8b5efc thru 0x8b7699 (0x179e bytes) 000463[0] 1788930440.334898 trunc from 0x8b769a to 0x3e284c (0x4d4e4d bytes) 000464[1] 1788930440.399633 trunc from 0x3e284d to 0x02b91f (0x3b6f2d bytes) 000465[1] 1788930440.462339 punch from 0x014eeb to 0x015b89 (0xc9f bytes) 000466[1] 1788930440.468255 punch from 0x00dea2 to 0x00f7c1 (0x1920 bytes) 000469[1] 1788930440.509177 punch from 0x0262a0 to 0x026c92 (0x9f3 bytes) 000471[1] 1788930440.538057 write 0x96df14 thru 0x96fd5d (0x1e4a bytes) 000472[0] 1788930440.547299 trunc from 0x96fd5e to 0x4ed53c (0x482821 bytes) 000473[1] 1788930440.606649 mapread 0x0d3d61 thru 0x0d4a72 (0xd12 bytes) 000474[1] 1788930440.655595 punch from 0x3294a2 to 0x32a438 (0xf97 bytes) 000477[1] 1788930440.735522 mapwrite 0x8d2596 thru 0x8d2bd9 (0x644 bytes) 000485[1] 1788930440.914326 write 0x537796 thru 0x5396fd (0x1f68 bytes) 000487[0] 1788930440.970705 write 0x990604 thru 0x9920a1 (0x1a9e bytes) 000488[0] 1788930441.002958 write 0x3d1fab thru 0x3d382b (0x1881 bytes) 000489[1] 1788930441.028844 trunc from 0x9920a2 to 0x2eb17b (0x6a6f26 bytes) 000490[1] 1788930441.113428 read 0x282a1d thru 0x28459d (0x1b81 bytes) 000491[0] 1788930441.194635 punch from 0x1829ee to 0x183c43 (0x1256 bytes) 000492[1] 1788930441.244515 trunc from 0x2eb17c to 0x8db3e0 (0x5f0265 bytes) 000493[0] 1788930441.287454 mapwrite 0x9e2c90 thru 0x9e37a7 (0xb18 bytes) 000494[1] 1788930441.400638 read 0x8bf162 thru 0x8bf205 (0xa4 bytes) 000496[1] 1788930441.446292 mapread 0x19c93d thru 0x19d6da (0xd9e bytes) 000497[0] 1788930441.525443 punch from 0x2776b5 to 0x2780b2 (0x9fe bytes) 000499[0] 1788930441.609853 trunc from 0x9e37a8 to 0x533e4b (0x4af95c bytes) All operations completed A-OK! PASS 16b (38s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 01:07:28 (1788930448) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 01:07:33 (1788930453) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.627568 s, 13.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.324102 s, 25.9 MB/s ldlm.namespaces.MGC192.168.206.147@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9004854a0000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff90049805d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9004854a0000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9004854a0000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.33 s, 9.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.48918 s, 23.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.05019 s, 34.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.28957 s, 16.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.2135 s, 12.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.56621 s, 23.0 MB/s PASS 16d (53s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 01:08:27 (1788930507) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.515629 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158813 s, 6.6 MB/s PASS 16e (10s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 01:08:37 (1788930517) TIMEOUT PASS 16f (30s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 01:09:07 (1788930547) TIMEOUT PASS 16g (28s) == sanityn test 16h: mmap read after truncate file ======= 01:09:36 (1788930576) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (8s) == sanityn test 16i: read after truncate file ============ 01:09:44 (1788930584) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (7s) == sanityn test 16j: race dio with buffered i/o ========== 01:09:51 (1788930591) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.4557 s, 18.0 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.67692 s, 15.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.07509 s, 8.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.94993 s, 13.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.61717 s, 10.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.69335 s, 9.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.47314 s, 7.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.17617 s, 22.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.49099 s, 17.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.34714 s, 7.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.50299 s, 7.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.1509 s, 12.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.44415 s, 10.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.03946 s, 8.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.16984 s, 8.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.24744 s, 8.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.49918 s, 7.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.89173 s, 6.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.93781 s, 6.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (44s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 01:10:35 (1788930635) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!skipping zero size read : Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported : Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Wed Sep 9 01:11:29 EDT 2026 All operations completed A-OK! All operations completed A-OK! + fsx 2 done: rc=0 Wed Sep 9 01:11:30 EDT 2026 + fsx 3 done: rc=0 Wed Sep 9 01:11:30 EDT 2026 + fsx 4 done: rc=0 Wed Sep 9 01:11:30 EDT 2026 All operations completed A-OK! + fsx 5 done: rc=0 Wed Sep 9 01:11:30 EDT 2026 PASS 16k (66s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 01:11:42 (1788930702) fail_loc=0x8000030a PASS 17 (12s) == sanityn test 18: mmap sanity check =========================================================================================== 01:11:54 (1788930714) mmap test1: basic mmap operation (PASS, 0.21813s) mmap test2: MAP_PRIVATE not write back (PASS, 0.080285s) mmap test3: concurrent mmap ops on two nodes (PASS, 14.566s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.7396s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.2084s) mmap test6: check mmap write/read content on two nodes (PASS, 0.30271s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.1443s) mmap test11: multi-thread shared mmap (PASS, 1.679s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1342s) PASS 18 (34s) == sanityn test 19: test concurrent uncached read races ========================================================================= 01:12:28 (1788930748) oleg647-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg647-client: oleg647-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 01:12:33 (1788930753) PASS 20 (8s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 01:12:41 (1788930761) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (7s) == sanityn test 23: others should see updated atime while another read============================================================== 01:12:48 (1788930768) now is 1788930771 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 new atime is 1788930832 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 01:13:59 (1788930839) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 01:14:07 (1788930847) PASS 24b (7s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 01:14:14 (1788930854) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (8s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 01:14:22 (1788930862) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (4s) == sanityn test 26a: allow mtime to get older ============ 01:14:26 (1788930866) utime: good mknod times 1788930866 <= 1788930867 <= 1788930867 for /mnt/lustre/f26a utime: good mknod times 1788930866 <= 1788930867 <= 1788930867 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (7s) == sanityn test 26b: sync mtime between ost and mds ====== 01:14:34 (1788930874) PASS 26b (9s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 01:14:43 (1788930883) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-09 01:14:45.000000000 -0400 Birth: 2026-09-09 01:14:45.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-09 01:14:45.000000000 -0400 Birth: 2026-09-09 01:14:45.000000000 -0400 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 01:14:53 (1788930893) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 01:15:03 (1788930903) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 01:15:11 (1788930911) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg647-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.206.147@tcp:/lustre 7542784 38912 7496704 1% /mnt/lustre2 PASS 31b (25s) == sanityn test 31r: open-rename(replace) race =========== 01:15:36 (1788930936) -rw-r--r-- 1 root root 0 Sep 9 01:15 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 01:15:46 (1788930946) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43213 Terminated tail -f $tfile PASS 31s (7s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 01:15:54 (1788930954) PASS 31t (9s) == sanityn test 32b: lockless i/o ======================== 01:16:02 (1788930962) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (3s) == sanityn test 32c: contention detection ================ 01:16:06 (1788930966) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.75973 s, 29.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.81261 s, 28.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.30085 s, 22.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.7153 s, 19.3 MB/s contention_seconds 4, contention_events 22 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.08022 s, 25.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.41412 s, 21.7 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.43869 s, 36.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.4367 s, 36.5 MB/s PASS 32c (33s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 01:16:40 (1788931000) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (5s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 01:16:45 (1788931005) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 01:16:49 (1788931009) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 01:16:53 (1788931013) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (4s) == sanityn test 34: no lock timeout under IO ============= 01:16:57 (1788931017) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg647-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9004854a0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9004854a0000.ost_server_uuid in FULL state after 0 sec oleg647-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9004854a0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9004854a0000.ost_server_uuid in IDLE state after 0 sec oleg647-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9004854a0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9004854a0000.ost_server_uuid in FULL state after 0 sec oleg647-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9004854a0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9004854a0000.ost_server_uuid in IDLE state after 0 sec oleg647-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9004854a0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9004854a0000.ost_server_uuid in FULL state after 0 sec oleg647-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9004854a0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9004854a0000.ost_server_uuid in IDLE state after 0 sec PASS 34 (138s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 01:19:16 (1788931156) fail_loc=0x80000317 Race attempt 0 Wait for 50524 50566 for 60 sec... - open/close 685 (time 1788931172.97 total 10.02 last 68.39) - open/close 1395 (time 1788931182.98 total 20.03 last 70.90) - open/close 2079 (time 1788931192.99 total 30.04 last 68.35) - open/close 2751 (time 1788931202.99 total 40.04 last 67.17) - open/close 3489 (time 1788931213.00 total 50.05 last 73.71) total: 4000 open/close in 57.95 seconds: 69.03 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.206.147@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.206.147@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (73s) == sanityn test 36: handle ESTALE/open-unlink correctly == 01:20:28 (1788931228) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg647-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.87861 s, 27.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.60635 s, 32.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.46373 s, 35.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7495680) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.16938 s, 44.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7497728) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.44649 s, 36.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7493632) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.25945 s, 41.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.64161 s, 31.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7497728) after_dd(7442432) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.45928 s, 35.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7497728) after_dd(7442432) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.66186 s, 31.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7497728) after_dd(7442432) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.51638 s, 34.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7497728) after_dd(7442432) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.74967 s, 30.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7497728) after_dd(7442432) after(7493632) PASS 36 (334s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 01:26:02 (1788931562) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 - create 1278 (time 1788931576.58 total 10.01 last 127.72) - create 2309 (time 1788931586.58 total 20.01 last 103.06) - create 3371 (time 1788931596.59 total 30.01 last 106.17) - create 4641 (time 1788931606.59 total 40.02 last 126.96) - create 5830 (time 1788931616.59 total 50.02 last 118.88) - create 7175 (time 1788931626.59 total 60.02 last 134.49) - create 8532 (time 1788931636.60 total 70.02 last 135.64) - create 9893 (time 1788931646.61 total 80.03 last 135.94) total: 10000 create in 81.14 seconds: 123.24 ops/second PASS 37 (93s) == sanityn test 39a: file mtime does not change after rename ========================================================== 01:27:35 (1788931655) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 01:27:43 (1788931663) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 01:27:51 (1788931671) repeat after cancel_lru_locks PASS 39c (9s) == sanityn test 39d: sync write should update mtime ====== 01:28:00 (1788931680) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 01:28:09 (1788931689) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (18s) == sanityn test 40c: pdirops: link and others ======================================================================== 01:28:28 (1788931708) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (18s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 01:28:45 (1788931725) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (15s) == sanityn test 40e: pdirops: rename and others ======================================================================== 01:29:00 (1788931740) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (15s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 01:29:15 (1788931755) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (12s) == sanityn test 41b: pdirops: create vs create ======================================================================== 01:29:27 (1788931767) open: File exists Conflict PASS 41b (13s) == sanityn test 41c: pdirops: create vs link ======================================================================== 01:29:40 (1788931780) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 01:29:54 (1788931794) Conflict PASS 41d (12s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 01:30:06 (1788931806) Conflict PASS 41e (12s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 01:30:19 (1788931819) Conflict PASS 41f (12s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 01:30:32 (1788931832) Conflict PASS 41g (13s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 01:30:44 (1788931844) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (14s) == sanityn test 41i: reint_open: create vs create ======== 01:30:58 (1788931858) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1030s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 01:48:08 (1788932888) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (14s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 01:48:22 (1788932902) open: File exists Conflict PASS 42b (15s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 01:48:37 (1788932917) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (15s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 01:48:52 (1788932932) Conflict PASS 42d (15s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 01:49:08 (1788932948) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (13s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 01:49:21 (1788932961) Conflict PASS 42f (14s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 01:49:35 (1788932975) Conflict PASS 42g (13s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 01:49:48 (1788932988) Conflict PASS 42h (18s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 01:50:06 (1788933006) PASS 43a (82s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 01:51:28 (1788933088) Conflict PASS 43b (14s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 01:51:42 (1788933102) Conflict PASS 43c (14s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 01:51:56 (1788933116) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (13s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 01:52:09 (1788933129) Conflict PASS 43e (17s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 01:52:26 (1788933146) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (16s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 01:52:42 (1788933162) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (15s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 01:52:57 (1788933177) Conflict PASS 43h (14s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 01:53:12 (1788933192) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 01:53:15 (1788933195) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (138s) == sanityn test 43k: unlink vs create ==================== 01:55:33 (1788933333) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1214s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 02:15:47 (1788934547) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Wed Sep 9 02:15:56 EDT 2026 PASS 44a (14s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 02:16:01 (1788934561) open: File exists Conflict PASS 44b (14s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 02:16:15 (1788934575) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (14s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 02:16:29 (1788934589) Conflict PASS 44d (16s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 02:16:45 (1788934605) Conflict PASS 44e (15s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 02:17:00 (1788934620) Conflict PASS 44f (15s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 02:17:15 (1788934635) Conflict PASS 44g (13s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 02:17:28 (1788934648) Conflict PASS 44h (14s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 02:17:42 (1788934662) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 02:17:46 (1788934666) PASS 45a (129s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 02:19:55 (1788934795) Conflict PASS 45b (13s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 02:20:08 (1788934808) Conflict PASS 45c (14s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 02:20:22 (1788934822) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (15s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 02:20:37 (1788934837) Conflict PASS 45e (15s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 02:20:52 (1788934852) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (14s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 02:21:06 (1788934866) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (13s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 02:21:19 (1788934879) Conflict PASS 45h (15s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 02:21:34 (1788934894) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 02:21:37 (1788934897) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1205s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 02:41:42 (1788936102) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (13s) == sanityn test 46b: pdirops: link vs create ======================================================================== 02:41:55 (1788936115) open: File exists Conflict PASS 46b (11s) == sanityn test 46c: pdirops: link vs link ======================================================================== 02:42:06 (1788936126) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (12s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 02:42:19 (1788936139) Conflict PASS 46d (13s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 02:42:33 (1788936153) Conflict PASS 46e (14s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 02:42:47 (1788936167) Conflict PASS 46f (18s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 02:43:05 (1788936185) Conflict PASS 46g (12s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 02:43:17 (1788936197) Conflict PASS 46h (14s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 02:43:31 (1788936211) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 02:43:35 (1788936215) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (2s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 02:43:38 (1788936218) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 02:43:41 (1788936221) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (2s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 02:43:43 (1788936223) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 02:43:46 (1788936226) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 02:43:49 (1788936229) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (2s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 02:43:52 (1788936232) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (2s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 02:43:54 (1788936234) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.169793 s, 60.3 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (12s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 02:44:06 (1788936246) 0+1 records in 0+1 records out 158 bytes copied, 0.0323585 s, 4.9 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218909) - No such process PASS 51a (10s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 02:44:16 (1788936256) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0106065 s, 96.5 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322246 s, 32.5 MB/s 3145728 PASS 51b (19s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 02:44:35 (1788936275) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0280436 s, 36.5 kB/s PASS 51c (12s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 02:44:48 (1788936288) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546318 s, 19.2 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (8s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 02:44:56 (1788936296) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 read lease(1) has applied. PASS 51e (9s) == sanityn test 54: rename locking ======================= 02:45:05 (1788936305) llite.lustre-ffff9004854a0000.xattr_cache=1 llite.lustre-ffff90049805d000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 02:45:07.000000000 -0400 Modify: 2026-09-09 02:45:07.000000000 -0400 Change: 2026-09-09 02:45:07.000000000 -0400 Birth: 2026-09-09 02:45:07.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 02:45:14.000000000 -0400 Modify: 2026-09-09 02:45:15.000000000 -0400 Change: 2026-09-09 02:45:15.000000000 -0400 Birth: 2026-09-09 02:45:14.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 02:45:20.000000000 -0400 Modify: 2026-09-09 02:45:21.000000000 -0400 Change: 2026-09-09 02:45:21.000000000 -0400 Birth: 2026-09-09 02:45:21.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 02:45:20.000000000 -0400 Modify: 2026-09-09 02:45:22.000000000 -0400 Change: 2026-09-09 02:45:22.000000000 -0400 Birth: 2026-09-09 02:45:21.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 02:45:27.000000000 -0400 Modify: 2026-09-09 02:45:28.000000000 -0400 Change: 2026-09-09 02:45:28.000000000 -0400 Birth: 2026-09-09 02:45:27.000000000 -0400 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 02:45:27.000000000 -0400 Modify: 2026-09-09 02:45:28.000000000 -0400 Change: 2026-09-09 02:45:28.000000000 -0400 Birth: 2026-09-09 02:45:27.000000000 -0400 PASS 54 (34s) == sanityn test 55a: rename vs unlink target dir ========= 02:45:39 (1788936339) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (13s) == sanityn test 55b: rename vs unlink source dir ========= 02:45:52 (1788936352) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (14s) == sanityn test 55c: rename vs unlink orphan target dir == 02:46:06 (1788936366) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (18s) == sanityn test 55d: rename file vs link ================= 02:46:24 (1788936384) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (15s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 02:46:39 (1788936399) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (2s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 02:46:42 (1788936402) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (20s) == sanityn test 55g: rename: race with trylock =========== 02:47:01 (1788936421) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (22s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 02:47:23 (1788936443) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788936445 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.581324 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (113s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 02:49:16 (1788936556) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788936560 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 0.939032 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (142s) == sanityn test 60: Verify data_version behaviour ======== 02:51:38 (1788936698) PASS 60 (11s) == sanityn test 70a: cd directory && rm directory ======== 02:51:49 (1788936709) PASS 70a (11s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 02:52:00 (1788936720) PASS 70b (15s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 02:52:15 (1788936735) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00848596 s, 7.7 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (5s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 02:52:20 (1788936740) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0157775 s, 4.2 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (5s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 02:52:25 (1788936745) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (5s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 02:52:30 (1788936750) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0480118 s, 1.4 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (4s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 02:52:34 (1788936754) llite.lustre-ffff9004854a0000.xattr_cache=1 llite.lustre-ffff90049805d000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (7s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 02:52:42 (1788936762) llite.lustre-ffff9004854a0000.xattr_cache=1 llite.lustre-ffff90049805d000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (7s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 02:52:48 (1788936768) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0133666 s, 76.6 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0117244 s, 87.3 kB/s 231719: get lock2 231720: get lock1 231720: done 231720: get lock2 231719: done 231719: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 231719: cannot get lock1: Resource deadlock avoided 231720: done 231720: exit rc=0 231719: done 231719: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 02:52:59 (1788936779) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.107463 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608096 s, 17.2 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232300 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (13s) == sanityn test 76: Verify MDT open_files listing ======== 02:53:12 (1788936792) opening files: ........ closing files: ........ PASS 76 (73s) == sanityn test 77a: check FIFO NRS policy =============== 02:54:25 (1788936865) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.561743 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579777 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821511 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154607 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226592 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247808 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254492 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332196 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215493 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244909 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266152 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256918 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319871 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318287 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277564 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36882 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.463148 s, 2.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927838 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477204 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952111 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303275 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667232 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235656 s, 44.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314716 s, 33.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105183 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150027 s, 69.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283491 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172409 s, 60.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034551 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101662 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10308 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364579 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251873 s, 41.6 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (11s) == sanityn test 77b: check CRR-N NRS policy ============== 02:54:36 (1788936876) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.510283 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10145 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114373 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123859 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049301 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226317 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967948 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975794 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269232 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290728 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262576 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313315 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147903 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257448 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327509 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3509 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100259 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571439 s, 18.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847161 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108573 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851629 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954605 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072968 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706494 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848633 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11259 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948813 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133875 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804053 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103172 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958612 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110108 s, 9.5 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg647-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg647-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.526978 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393195 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493559 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874643 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880125 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896643 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350231 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11935 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905531 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160891 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14784 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239511 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297793 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308766 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322885 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313963 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401582 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527087 s, 19.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671711 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111879 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386237 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567553 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060251 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109191 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115102 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319325 s, 32.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407014 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493857 s, 21.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607524 s, 17.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516079 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932334 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567928 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601391 s, 17.4 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (14s) == sanityn test 77c: check ORR NRS policy ================ 02:54:50 (1788936890) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.425191 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505014 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100974 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165315 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144973 s, 7.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17551 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194343 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189963 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308086 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311905 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289271 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263665 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245531 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251232 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238216 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244944 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341919 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302971 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289342 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658961 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337141 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657175 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381218 s, 27.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790162 s, 13.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814121 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811173 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760065 s, 13.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810135 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543154 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556445 s, 18.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847258 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103188 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824123 s, 12.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg647-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg647-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.536497 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502968 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492167 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869931 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871166 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111133 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187896 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215916 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223795 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240354 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196232 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222332 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203703 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258195 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25699 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273126 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278434 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604604 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103501 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673274 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430566 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056652 s, 18.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115902 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474752 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582967 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064473 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082459 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868994 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717047 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626473 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163697 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10297 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105541 s, 9.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.382756 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804105 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065984 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193787 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251485 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244531 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283298 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284989 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334849 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332456 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380443 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402115 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351982 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374903 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.435085 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41493 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.423035 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886683 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274045 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101955 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14367 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108967 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552781 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182248 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103472 s, 101 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251599 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247548 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107008 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02565 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248114 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780438 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527076 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643422 s, 16.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=2 removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (17s) == sanityn test 77d: check TRR nrs policy ================ 02:55:07 (1788936907) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.743352 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920939 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155084 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146188 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237387 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237117 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373008 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373357 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377735 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.423064 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431804 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.427086 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.460937 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409948 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396437 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394576 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.476084 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325899 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00676855 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337852 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817265 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478312 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379332 s, 27.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558025 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458866 s, 22.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123706 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468338 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496598 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539154 s, 19.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574778 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599172 s, 17.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079253 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970122 s, 10.8 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg647-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg647-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.385914 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081157 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822249 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150816 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167744 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209484 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258497 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289142 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288123 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242475 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278183 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298947 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299187 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292348 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296327 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274596 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295378 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498668 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903729 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940353 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499083 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594824 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675464 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102049 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735611 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765954 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115845 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838372 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128152 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094067 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149172 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107629 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149746 s, 7.0 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg647-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg647-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.526908 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901028 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796995 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795473 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943301 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184126 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238208 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275771 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254441 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34446 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254151 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339379 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312204 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35993 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329392 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306972 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381073 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222315 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00441833 s, 237 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00404463 s, 259 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00628423 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00311392 s, 337 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598148 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113319 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611567 s, 17.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837175 s, 12.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016642 s, 63.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523263 s, 20.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107619 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565794 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589703 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493492 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478805 s, 21.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (17s) == sanityn test 77e: check TBF NID nrs policy ============ 02:55:24 (1788936924) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.206.47@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.470414 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719561 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664913 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086395 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468468 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150681 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254743 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271074 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365974 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204229 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336115 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.415037 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307944 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41973 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213517 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236805 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312829 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634173 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109431 s, 95.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660348 s, 15.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666044 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110355 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783525 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722516 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790591 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861595 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360381 s, 29.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913302 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103278 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530548 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101109 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110782 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117574 s, 8.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.519723 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108547 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143952 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908205 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207869 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123485 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179009 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275428 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289661 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194577 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254518 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253978 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294421 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283984 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243498 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291117 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233158 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605761 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390752 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662214 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440812 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760912 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525833 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111382 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106924 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877914 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928938 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947198 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127582 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823272 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108137 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084568 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878271 s, 11.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.564034 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086224 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103217 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14635 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181251 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223206 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209072 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221091 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255996 s, 4.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126423 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315845 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377762 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296485 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374359 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302258 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34003 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.513486 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111098 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00560418 s, 187 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102391 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916254 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549021 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715461 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142339 s, 73.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624204 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056581 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711719 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152986 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690866 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480711 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516652 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444461 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122509 s, 8.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg647-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg647-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.506094 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767398 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144021 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176801 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102103 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195854 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116752 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214832 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226915 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23501 s, 4.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286675 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217081 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257804 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359967 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266642 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410044 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468133 s, 2.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056322 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055475 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351966 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908261 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049682 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492082 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501117 s, 20.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933085 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802135 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579001 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954916 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131242 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078838 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104441 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962822 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967107 s, 10.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg647-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg647-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (27s) == sanityn test 77f: check TBF JobID nrs policy ========== 02:55:51 (1788936951) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg647-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.436816 s, 38.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916535 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903676 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231614 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241057 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181672 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247655 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219937 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254251 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259503 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281407 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24755 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312372 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365479 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331896 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361665 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378664 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170106 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141953 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953466 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841196 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826129 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746235 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0058878 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725498 s, 14.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693137 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636238 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579216 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111669 s, 93.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381746 s, 27.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477822 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337711 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331469 s, 31.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg647-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.47262 s, 35.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928277 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942034 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301076 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20449 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206949 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26835 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305433 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218172 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205969 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350208 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306976 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245862 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340945 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382932 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317146 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419802 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091247 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951337 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162077 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170172 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112286 s, 9.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112671 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139249 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115342 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128701 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130056 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131718 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198408 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147866 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135982 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150495 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140262 s, 7.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg647-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg647-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg647-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.673445 s, 24.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546942 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993376 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161868 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100564 s, 10.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15378 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123869 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151701 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164448 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229234 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189616 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223458 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189712 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351616 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00436994 s, 240 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32892 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00948376 s, 111 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372888 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00383045 s, 274 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.481664 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00251489 s, 417 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00912114 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00421716 s, 249 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220862 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191474 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00572276 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207008 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193403 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00408125 s, 257 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319122 s, 32.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141708 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310305 s, 33.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104147 s, 101 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg647-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.433045 s, 38.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548698 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098828 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118719 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11051 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11207 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934315 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996911 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109153 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172676 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113567 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096479 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118595 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184995 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164608 s, 6.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178636 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15041 s, 7.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125748 s, 83.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143273 s, 73.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179848 s, 58.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00579922 s, 181 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646433 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274417 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00742125 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00271038 s, 387 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302529 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00617764 s, 170 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00867576 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00473261 s, 222 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037778 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04477 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406112 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944855 s, 11.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (24s) == sanityn test 77g: Change TBF type directly ============ 02:56:15 (1788936975) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (15s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 02:56:30 (1788936990) oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg647-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg647-client: oleg647-server: ssh exited with exit code 2 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg647-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg647-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg647-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 PASS 77h (14s) == sanityn test 77i: Change rank of TBF rule ============= 02:56:44 (1788937004) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (26s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 02:57:10 (1788937030) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.86519 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525462 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817068 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838076 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143793 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159916 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159629 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125113 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664883 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539525 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158774 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29758 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295153 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162149 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181463 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20186 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207312 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364486 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387614 s, 27.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753767 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517375 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438589 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10506 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527589 s, 19.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586127 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094245 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055897 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566048 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112443 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597618 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102942 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669213 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686066 s, 15.3 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg647-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg647-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.99245 s, 13.1 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.654 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (66s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 02:58:16 (1788937096) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg647-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg647-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg647-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg647-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg647-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.3141 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10817 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019378 s, 54.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152866 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120712 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106645 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205568 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2036 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227782 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18938 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278114 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197813 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206549 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344934 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245515 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247336 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275818 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720177 s, 14.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738896 s, 14.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838495 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850117 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579813 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898412 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092233 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06645 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918194 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883897 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925989 s, 11.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109579 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115645 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982447 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100339 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126934 s, 8.3 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0177 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9395 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg647-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg647-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg647-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg647-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg647-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30066 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136217 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113891 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156322 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198571 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2103 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630326 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172577 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219092 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173263 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186255 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146236 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13777 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269921 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177475 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249623 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18849 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984269 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838145 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101679 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949953 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108108 s, 9.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097322 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416317 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827357 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673104 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582229 s, 18.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960597 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127003 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663609 s, 15.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080554 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104821 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967105 s, 10.8 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0419 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9909 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (153s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 03:00:49 (1788937249) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg647-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg647-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg647-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg647-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg647-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27792 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108105 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105914 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109539 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120053 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156072 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127766 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207932 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248321 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165953 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199117 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290945 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239957 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221359 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209562 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262382 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298047 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120178 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107528 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123967 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113298 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102222 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107197 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133356 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132081 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841079 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952188 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985323 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121018 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10689 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124548 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141072 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134277 s, 7.8 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0137 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9545 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg647-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg647-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg647-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg647-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg647-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30687 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901872 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110927 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801135 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149866 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251301 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280139 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28616 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262805 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318675 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291112 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345424 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212236 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380581 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3847 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412474 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.458135 s, 2.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845286 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14117 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512075 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159944 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102961 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650371 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144242 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561079 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145581 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173485 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468345 s, 22.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527228 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145975 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628722 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536558 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157728 s, 6.6 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg647-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg647-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9022 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9588 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (158s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 03:03:27 (1788937407) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg647-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.773355 s, 21.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118455 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193327 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113501 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359892 s, 29.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209846 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318475 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310261 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102704 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371269 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385426 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19834 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956974 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727925 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248073 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179266 s, 5.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146415 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164428 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109867 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103762 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094415 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110457 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121973 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153047 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143804 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186247 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183775 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191586 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134362 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161532 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166264 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202121 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194795 s, 5.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.946 s, 7.5 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0284 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.206.47@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.206.47@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.59175 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157221 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176915 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23098 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126997 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297673 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30641 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303432 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28256 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234351 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307832 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392672 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.405057 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.461454 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.488767 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.490037 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348216 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108444 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072193 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562822 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807212 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15759 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651281 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13121 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756744 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777704 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848199 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844763 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946528 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11808 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102258 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106557 s, 9.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109362 s, 9.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.71677 s, 12.0 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96814 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.206.47@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg647-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.513573 s, 32.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548289 s, 19.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101661 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105666 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105123 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773592 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269664 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227136 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248357 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161789 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197941 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195697 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195496 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30967 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233232 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289155 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.479144 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837437 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729131 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951722 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107614 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506042 s, 20.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072379 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630633 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113036 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068342 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497542 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700111 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866897 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926642 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921153 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748207 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111566 s, 9.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg647-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg647-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89122 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.3042 s, 19.8 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg647-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.587929 s, 28.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056639 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808403 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696302 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616599 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931846 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891095 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121479 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18073 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976305 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196085 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294221 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171074 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178605 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177676 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344321 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222548 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166007 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143556 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146874 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151701 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145856 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930189 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214403 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147841 s, 7.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127247 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146866 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143596 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210233 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179622 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184845 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172153 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182007 s, 5.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.5 s, 9.1 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94953 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.94448 s, 13.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97134 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg647-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.04405 s, 16.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751843 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082698 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121387 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208823 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24518 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155927 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284028 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285843 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307022 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279713 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353545 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365668 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338046 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364611 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368532 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394922 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281307 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104183 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777797 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462157 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106487 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498372 s, 21.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824534 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407708 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834822 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485466 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528606 s, 19.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530261 s, 19.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533936 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601291 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659255 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701045 s, 15.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg647-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg647-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7703 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6024 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg647-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.67273 s, 24.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671358 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683529 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146798 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216478 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241222 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256447 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306831 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186555 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241978 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190655 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268717 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259238 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321009 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3798 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337155 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335337 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190897 s, 54.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239187 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419127 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027906 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379244 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347644 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941068 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369655 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809259 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556819 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645763 s, 16.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142618 s, 73.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672223 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102656 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060739 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733116 s, 14.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5378 s, 10.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.007 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg647-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.628434 s, 26.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102732 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102134 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980238 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108255 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830203 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123035 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166921 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206266 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149182 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020572 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378949 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162078 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270658 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173273 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343923 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363769 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736152 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779088 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846948 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152456 s, 68.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156346 s, 67.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025188 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341834 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918816 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10701 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00236726 s, 443 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00257073 s, 408 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00574758 s, 182 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00721533 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555967 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173556 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199856 s, 52.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.6727 s, 9.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0571 s, 10.4 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.9285 s, 8.8 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0483 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (457s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 03:11:05 (1788937865) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg647-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.707314 s, 23.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723456 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872805 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126084 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180623 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177258 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146618 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169493 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175176 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131651 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310226 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33874 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310801 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331795 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377299 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352392 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288645 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791841 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672761 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853109 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698837 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608141 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076949 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800624 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844835 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769521 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716384 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900479 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839731 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767624 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137492 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954699 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104916 s, 10.0 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg647-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg647-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.43227 s, 12.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99335 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (55s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 03:11:59 (1788937919) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.206.47@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.206.47@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.653363 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889294 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135826 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140248 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166956 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857072 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128152 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156751 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189137 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102689 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175937 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278955 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242502 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21495 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316754 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314899 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352645 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193431 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204485 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197129 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17848 s, 5.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215777 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145332 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223952 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183395 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193318 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221197 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208496 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203212 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253827 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220589 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263562 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209173 s, 5.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=7 removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg647-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg647-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.83555 s, 11.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0174 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (56s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 03:12:55 (1788937975) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.206.47@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg647-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.16578 s, 14.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670262 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888226 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215643 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161076 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221815 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250367 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335488 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297552 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368777 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291504 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388677 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408848 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36705 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390629 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385412 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363683 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181641 s, 57.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759039 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516776 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579155 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060475 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455396 s, 23.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659992 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775136 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751362 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638444 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116954 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899744 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856731 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934467 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00571536 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123076 s, 8.5 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.56466 s, 13.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.38483 s, 19.5 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (50s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 03:13:45 (1788938025) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg647-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.710874 s, 23.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870338 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962026 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099471 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134128 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105898 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887793 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143861 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219951 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166496 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141277 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220049 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114642 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405919 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100888 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272717 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231507 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604143 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601275 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588086 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588672 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263906 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380032 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00609261 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462277 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303288 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00805744 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00554201 s, 189 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738091 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112519 s, 93.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717955 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126673 s, 82.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010008 s, 105 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg647-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg647-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1248 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0027 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.22076 s, 11.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94818 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (98s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 03:15:23 (1788938123) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg647-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.13494 s, 14.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550978 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124615 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11131 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111941 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107579 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140716 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289158 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264857 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324787 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337529 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280603 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273424 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280165 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352323 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251529 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316722 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137731 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125719 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139582 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065502 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138606 s, 75.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559755 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846431 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167556 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108569 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184294 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941818 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943183 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205891 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139426 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132732 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23 s, 4.6 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7864 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9586 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (76s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 03:16:40 (1788938200) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg647-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.65291 s, 25.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453195 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069005 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141822 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989319 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243503 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187464 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180185 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216427 s, 4.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218207 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145936 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271798 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206495 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268903 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24609 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193038 s, 5.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90049805d000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff90049805d000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255717 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328303 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359766 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329762 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583631 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913683 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064657 s, 16.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728434 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531807 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548937 s, 19.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513575 s, 20.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568108 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801321 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654009 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687221 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702561 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712923 s, 14.7 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.00847 s, 13.1 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89223 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg647-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.747069 s, 22.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768352 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649916 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938657 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202217 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238803 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17668 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141365 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277533 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173431 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293908 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171192 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409542 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161802 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289751 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.458428 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200293 s, 5.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281907 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745767 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056468 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320723 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349045 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355348 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119812 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465877 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548576 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623358 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625849 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621768 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719862 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647669 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078872 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824563 s, 12.7 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.8497 s, 8.8 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97891 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.0121 s, 9.5 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96611 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (147s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 03:19:08 (1788938348) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg647-client.virtnet /mnt/lustre (opts:) Stopping client oleg647-client.virtnet /mnt/lustre opts: Stopping clients: oleg647-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg647-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg647-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg647-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg647-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg647-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.47,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg647-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.47,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg647-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.47,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69873/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.47,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg647-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg647-server' oleg647-server: oleg647-server.virtnet: executing load_modules_local oleg647-server: kmemleak disabled oleg647-server: Loading modules from /home/green/git/lustre-release/lustre oleg647-server: detected 4 online CPUs by sysfs oleg647-server: MODOPTS_LIBCFS= oleg647-server: Force libcfs to create 2 CPU partitions oleg647-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre Starting client oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre Started clients oleg647-client.virtnet: 192.168.206.147@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre2 Starting client oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre2 Started clients oleg647-client.virtnet: 192.168.206.147@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff900490566800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9004989ea000.idle_timeout=debug osc.lustre-OST0001-osc-ffff900490566800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9004989ea000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.476421 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077795 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116659 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10362 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564674 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834588 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432929 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502902 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446208 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120221 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241634 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226878 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111632 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296614 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296876 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328384 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37753 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662909 s, 15.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349992 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483404 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669316 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888067 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862207 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640218 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535619 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492121 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056191 s, 18.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105347 s, 10.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647442 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652091 s, 16.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654683 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681967 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746631 s, 14.0 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.49285 s, 11.0 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.3065 s, 19.8 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.576758 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984016 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153385 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729968 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136241 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153229 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10235 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657952 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188476 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217938 s, 4.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23745 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156287 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197558 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267826 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212384 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276182 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144441 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356891 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785792 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167128 s, 62.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262288 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476934 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401753 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190798 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00228954 s, 458 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174566 s, 60.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150385 s, 69.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00999233 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00673607 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00773106 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193222 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127142 s, 82.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323513 s, 32.4 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg647-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg647-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.57235 s, 13.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88929 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (228s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 03:22:55 (1788938575) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg647-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (19s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 03:23:14 (1788938594) On MGS 192.168.206.147, active = nodemap.active=1 On MGS 192.168.206.147, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.642145 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987314 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102175 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775636 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184501 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206774 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185866 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155384 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185463 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236025 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217032 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177003 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151583 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26915 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217752 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307297 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198812 s, 5.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff900490566800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff900490566800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.517645 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.545957 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.538034 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.526996 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.5279 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.54251 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.538566 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.551651 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.594282 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.575646 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.580564 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.585258 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.605282 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.608205 s, 1.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.614863 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.619624 s, 1.7 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.95061 s, 11.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.40425 s, 19.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.206.147, TBF.id = On MGS 192.168.206.147, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.737793 s, 22.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795038 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118755 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064198 s, 16.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141202 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16914 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132957 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255662 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242206 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176281 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24641 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291276 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218462 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26942 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263728 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240081 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff900490566800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff900490566800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239137 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221481 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557325 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329846 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605839 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345354 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407365 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428861 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513452 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115933 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719982 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116526 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772718 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775786 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892064 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897754 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915898 s, 11.4 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.40436 s, 14.2 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.58058 s, 40.6 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg647-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg647-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 On MGS 192.168.206.147, active = nodemap.active=1 On MGS 192.168.206.147, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.518683 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618301 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746806 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11503 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255129 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267319 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181746 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275093 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239214 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215988 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223852 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178096 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251015 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242351 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237667 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192957 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233358 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff900490566800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff900490566800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14342 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147974 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151072 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228365 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164517 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173571 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175998 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177526 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189427 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200522 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192305 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff900490566800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267382 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225069 s, 4.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228953 s, 4.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234688 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240809 s, 4.4 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg647-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg647-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.36793 s, 12.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0848 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.206.147, TBF.id = On MGS 192.168.206.147, active = nodemap.active=0 PASS 77kj (180s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 03:26:14 (1788938774) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (10s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 03:26:24 (1788938784) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg647-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.36878 s, 311 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg647-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.1146 s, 261 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (59s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 03:27:23 (1788938843) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.766821 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099485 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184355 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210902 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253308 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316445 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113863 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140049 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196363 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29863 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212386 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409965 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39938 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.531874 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.544322 s, 1.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.710587 s, 1.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff900490566800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff900490566800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13334 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04053 s, 25.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365366 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815358 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368037 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141639 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889095 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186261 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126669 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205927 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944683 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149085 s, 7.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.667112 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196563 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15708 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.513815 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00743325 s, 141 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg647-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg647-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.57 s, 9.9 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.22916 s, 20.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.01477 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560587 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558719 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116175 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130819 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170511 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168403 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263479 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234035 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191665 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240348 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38206 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249761 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422389 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352347 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302978 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320192 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955683 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566812 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704982 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811974 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757017 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819968 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832235 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129309 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118733 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134356 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147792 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125322 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111216 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156863 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114546 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149865 s, 7.0 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg647-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg647-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.1834 s, 8.6 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.44231 s, 19.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (100s) == sanityn test 77o: Changing rank should not panic ====== 03:29:03 (1788938943) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (12s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 03:29:16 (1788938956) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (131s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 03:31:26 (1788939086) mds.MDS.mdt.nrs_policies=tbf oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg647-client: oleg647-server: ssh exited with exit code 36 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg647-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg647-client: oleg647-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (42s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 03:32:08 (1788939128) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273581 kill md_thread xargs PASS 77r (62s) == sanityn test 77s: Check TBF LRU shrinker ============== 03:33:11 (1788939191) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 8 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 8 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (61s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 03:34:11 (1788939251) oleg647-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg647-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg647-client: oleg647-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (10s) == sanityn test 79: xattr: intent error ================== 03:34:22 (1788939262) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935647 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 03:34:24.000000000 -0400 Modify: 2026-09-09 03:34:24.000000000 -0400 Change: 2026-09-09 03:34:24.000000000 -0400 Birth: 2026-09-09 03:34:25.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (20s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 03:34:41 (1788939281) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (4s) == sanityn test 80b: Accessing directory during migration ========================================================== 03:34:45 (1788939285) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (4s) == sanityn test 81a: rename and stat under striped directory ========================================================== 03:34:49 (1788939289) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 03:34:53 (1788939293) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 03:34:56 (1788939296) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 03:34:59 (1788939299) - open/close 742 (time 1788939314.28 total 10.00 last 74.18) - open/close 1242 (time 1788939324.29 total 20.02 last 49.93) - open/close 1767 (time 1788939334.30 total 30.02 last 52.50) - open/close 2303 (time 1788939344.30 total 40.02 last 53.59) total: 2500 open/close in 43.17 seconds: 57.91 ops/second snapshot_time 1788939438.555200057 secs.nsecs start_time 1788939302.331252636 secs.nsecs elapsed_time 136.223947421 secs.nsecs open 2505 samples [usecs] 131 22460 4897603 21683701719 close 2505 samples [usecs] 150 8765 961328 959172832 mknod 2501 samples [usecs] 570 22306 4208294 16993672496 unlink 2500 samples [usecs] 747 90749 5926974 66349043878 mkdir 2 samples [usecs] 1610 2967 4577 11395189 rmdir 2 samples [usecs] 911 5809 6720 34574402 rename 2500 samples [usecs] 485 56789 16448200 227139146416 getattr 3762 samples [usecs] 10 11456 287354 592713328 parallel_rename_file 2500 samples [usecs] 485 56789 16448200 227139146416 crossdir_rename 2500 samples [usecs] 485 56789 16448200 227139146416 readpage 21 samples [usecs] 6848 54494 483962 15553794284 PASS 81d (146s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 03:37:25 (1788939445) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7630 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7630 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 03:37:32 (1788939452) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 03:37:36 (1788939456) fail_loc=0x8000060b PASS 84 (15s) == sanityn test 85: Lustre API root cache race =========== 03:37:51 (1788939471) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788939473 Finishing test test1 at 1788939477 PASS 85 (12s) == sanityn test 90: open/create and unlink striped directory ========================================================== 03:38:03 (1788939483) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 03:38:06 (1788939486) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 03:38:10 (1788939490) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 03:38:14 (1788939494) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5761 0x1681 0x240000400 1 5722 0x165a 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5723 0x165b 0x280000400 0 5762 0x1682 0x240000400 rc1=0 and rc2=0 PASS 93 (17s) == sanityn test 94: signal vs CP callback race =========== 03:38:30 (1788939510) osc.lustre-OST0000-osc-ffff900490566800.idle_timeout=0 osc.lustre-OST0000-osc-ffff9004989ea000.idle_timeout=0 osc.lustre-OST0001-osc-ffff900490566800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9004989ea000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.220392 s, 37.2 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.03534 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 290370 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0477704 s, 85.7 kB/s osc.lustre-OST0000-osc-ffff900490566800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9004989ea000.idle_timeout=debug osc.lustre-OST0001-osc-ffff900490566800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9004989ea000.idle_timeout=debug PASS 94 (15s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 03:38:45 (1788939525) llite.lustre-ffff900490566800.fast_read=0 llite.lustre-ffff9004989ea000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0870403 s, 24.1 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff900490566800.fast_read=1 llite.lustre-ffff9004989ea000.fast_read=1 PASS 95a (20s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 03:39:05 (1788939545) llite.lustre-ffff900490566800.fast_read=0 llite.lustre-ffff9004989ea000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00743292 s, 1.7 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff900490566800.fast_read=1 llite.lustre-ffff9004989ea000.fast_read=1 PASS 95b (14s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 03:39:19 (1788939559) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 03:39:22 (1788939562) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0150524 s, 272 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (6s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 03:39:28 (1788939568) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.171081 s, 12.3 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (7s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 03:39:35 (1788939575) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.148822 s, 14.1 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (7s) == sanityn test 100e: DoM: read on open and file size ==== 03:39:42 (1788939582) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00550364 s, 2.2 MB/s PASS 100e (5s) == sanityn test 101a: Discard DoM data on unlink ========= 03:39:48 (1788939588) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00878013 s, 467 kB/s PASS 101a (6s) == sanityn test 101b: Discard DoM data on rename ========= 03:39:54 (1788939594) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0144286 s, 284 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 03:40:01 (1788939601) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0242942 s, 169 kB/s PASS 101c (8s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 03:40:09 (1788939609) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 40 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf40:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (6s) == sanityn test 103: Test size correctness with lockahead ========================================================== 03:40:16 (1788939616) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788939619 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788939622 Finishing test test23 at 1788939622 PASS 103 (15s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 03:40:30 (1788939630) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 03:40:33 (1788939633) fail_loc=0x416 PASS 105 (27s) == sanityn test 106a: Verify the btime via statx() ======= 03:41:01 (1788939661) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 03:41:04 (1788939664) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0170001 s, 60.2 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938183 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 03:41:06.000000000 -0400 Modify: 2026-09-09 03:41:06.000000000 -0400 Change: 2026-09-09 03:41:06.000000000 -0400 Birth: 2026-09-09 03:41:06.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938183 -rw-r--r-- 1 0 0 1788939666 1788939666 1788939666 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938183 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 03:41:06.000000000 -0400 Modify: 2026-09-09 03:41:06.000000000 -0400 Change: 2026-09-09 03:41:06.000000000 -0400 Birth: 2026-09-09 03:41:06.000000000 -0400 1788939666 1024 1 PASS 106b (9s) == sanityn test 106c: Verify statx attributes mask ======= 03:41:13 (1788939673) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 03:41:21 (1788939681) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.353222 s, 29.7 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5730 0x1662 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 First grouplock blocks second one PASS 107a (10s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 03:41:31 (1788939691) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.369663 s, 28.4 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5771 0x168b 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (15s) == sanityn test 108a: lseek: parallel updates ============ 03:41:46 (1788939706) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0121024 s, 826 kB/s llite.lustre-ffff900490566800.hybrid_io=0 llite.lustre-ffff9004989ea000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.70772 s, 1.8 MB/s PASS 108a (13s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 03:42:00 (1788939720) 192.168.206.147@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg647-client.virtnet /mnt/lustre (opts:) 192.168.206.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg647-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre2 Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre 192.168.206.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.206.147@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg647-client.virtnet /mnt/lustre (opts:) Stopping client oleg647-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg647-server' oleg647-server: oleg647-server.virtnet: executing load_modules_local oleg647-server: kmemleak disabled oleg647-server: Loading modules from /home/green/git/lustre-release/lustre oleg647-server: detected 4 online CPUs by sysfs oleg647-server: MODOPTS_LIBCFS= oleg647-server: Force libcfs to create 2 CPU partitions oleg647-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre2 PASS 109 (50s) == sanityn test 110: do not grant another lock on resend ========================================================== 03:42:49 (1788939769) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 03:42:52.000000000 -0400 Modify: 2026-09-09 03:42:52.000000000 -0400 Change: 2026-09-09 03:42:52.000000000 -0400 Birth: 2026-09-09 03:42:52.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg647-client.virtnet oleg647-client.virtnet PASS 110 (74s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 03:44:03 (1788939843) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (3s) == sanityn test 112: update max-inherit in default LMV === 03:44:06 (1788939846) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 03:44:10 (1788939850) lustre-OST0000-osc-ffff900485353800 active. lustre-OST0001-osc-ffff900485353800 active. lustre-OST0000-osc-ffff900485353800 active. lustre-OST0001-osc-ffff900485353800 active. MGC192.168.206.147@tcp active. PASS 113 (8s) == sanityn test 114: implicit default LMV inherit ======== 03:44:18 (1788939858) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (4s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 03:44:22 (1788939862) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (5s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 03:44:27 (1788939867) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (5s) == sanityn test 121: trunc append race =================== 03:44:32 (1788939872) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0438855 s, 467 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.143927 s, 142 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (13s) == sanityn test 122: directory size is consistent across mounts ========================================================== 03:44:45 (1788939885) PASS 122 (11s) == sanityn test 200: service remains healthy while able to process request ========================================================== 03:44:56 (1788939896) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=8 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1545 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3719 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.4506 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.4696 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.4478 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.5313 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2069 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.5061 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.4746 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.5265 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.5578 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.6037 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3886 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.6611 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.7013 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.7127 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg647-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff900483ebf800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff900483ebf800.ost_server_uuid in IDLE state after 0 sec PASS 200 (88s) cleanup: ====================================================== == sanityn test complete, duration 10099 sec ============= 03:46:27 (1788939987) === sanityn: start cleanup 03:46:30 (1788939990) === Stopping clients: oleg647-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg647-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 03:50:30 (1788940230) ===