-----============= acceptance-small: sanityn ============----- Sat Sep 5 01:52:14 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg602-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 01:52:33 (1788587553) === Starting client oleg602-client.virtnet: -o user_xattr,flock 192.168.206.102@tcp:/lustre /mnt/lustre2 Mount client oleg602-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.206.102@tcp:/lustre /mnt/lustre2 Started clients oleg602-client.virtnet: 192.168.206.102@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg602-client.virtnet: executing check_config_client /mnt/lustre oleg602-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg602-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8b888624e800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8b8893a71800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b888624e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b8893a71800.idle_timeout=debug disable quota as required oleg602-server: oleg602-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 01:53:08 (1788587588) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051944 s, 20.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8428] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 01:53:10 (1788587590) PASS 0 (9s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 01:53:19 (1788587599) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (6s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 01:53:25 (1788587605) -rw-r--r-- 1 root root 0 Sep 5 01:53 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (7s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 01:53:32 (1788587612) -rw-r--r-- 1 root root 0 Sep 5 01:53 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 01:53:39 (1788587619) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 01:53:44 (1788587624) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (6s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 01:53:50 (1788587630) total 1 -rwxrwxrwx 1 root root 0 Sep 5 01:53 f2a -rwxrwxrwx 1 root root 0 Sep 5 01:53 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 5 01:53 f2a -rwxrwxrwx 1 root root 0 Sep 5 01:53 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0121465 s, 42.2 kB/s PASS 2e (7s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 01:53:57 (1788587637) /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has type file OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has perms 0777 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by user #500 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by group #500 OK Can't lstat /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn: No such file or directory PASS 2f (6s) == sanityn test 2g: check blocks update on sync write ==== 01:54:04 (1788587644) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.246275 s, 8.5 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (6s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 01:54:10 (1788587650) PASS 3 (6s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 01:54:16 (1788587656) PASS 4 (6s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 01:54:22 (1788587662) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 01:54:28 (1788587668) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (5s) == sanityn test 7: remove of open directory on other node ======================================================================= 01:54:33 (1788587673) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (7s) == sanityn test 8: remove of open special file on other node ==================================================================== 01:54:40 (1788587680) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 01:54:47 (1788587687) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 01:54:54 (1788587694) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427034 s, 24.6 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0548539 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (6s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 01:55:00 (1788587700) 1+0 records in 1+0 records out 1 byte copied, 0.00698921 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.017774 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0212038 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0100053 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0189565 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00842435 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00940248 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0117033 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0109456 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0294781 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0206274 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0185437 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 01:55:08 (1788587708) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0369492 s, 83.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0295759 s, 138 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000385008 s, 8.0 MB/s PASS 10b (6s) == sanityn test 11: execution of file opened for write should return error ============================================================== 01:55:14 (1788587714) striped dir -i1 -c2 -H crush /mnt/lustre/d11 multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8428 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 01:55:21 (1788587721) warning: '-runtime' deprecated, use '-t runtime' instead - link 369 (time 1788587733.39 total 10.04 last 36.74) - link 705 (time 1788587743.42 total 20.07 last 33.51) - link 1003 (time 1788587753.43 total 30.07 last 29.79) - link 1353 (time 1788587763.44 total 40.09 last 34.93) - link 1666 (time 1788587773.45 total 50.10 last 31.28) - link 1963 (time 1788587783.48 total 60.13 last 29.61) - link 2272 (time 1788587793.50 total 70.15 last 30.84) - link 2603 (time 1788587803.52 total 80.17 last 33.03) - link 2944 (time 1788587813.54 total 90.19 last 34.04) total: 3266 link in 100.01 seconds: 32.66 ops/second using seed 81225802 running for 100 seconds - stat 7761 (time 1788587735 ; total 11 ; last 11) - stat 10000 (time 1788587737 ; total 13 ; last 2) - stat 17634 (time 1788587748 ; total 24 ; last 11) - stat 20000 (time 1788587751 ; total 27 ; last 3) - stat 28354 (time 1788587762 ; total 38 ; last 11) - stat 30000 (time 1788587764 ; total 40 ; last 2) - stat 36346 (time 1788587775 ; total 51 ; last 11) - stat 39766 (time 1788587786 ; total 62 ; last 11) - stat 40000 (time 1788587786 ; total 62 ; last 0) - stat 42761 (time 1788587797 ; total 73 ; last 11) - stat 45766 (time 1788587808 ; total 84 ; last 11) - stat 48962 (time 1788587819 ; total 95 ; last 11) - stat 50000 (time 1788587823 ; total 99 ; last 4) total: 50427 stats in 100 seconds: 504.269989 stats/second - unlinked 0 (time 1788587774 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3266) error: No such file or directory total: 3265 unlinks in 84 seconds: 38.869049 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (22794) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (22797) - No such process PASS 12 (142s) == sanityn test 13: test directory page revocation ======= 01:57:43 (1788587863) striped dir -i1 -c2 -H crush2 /mnt/lustre/d13 f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 01:57:52 (1788587872) striped dir -i0 -c2 -H crush /mnt/lustre/d14aa.sanityn multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.8428 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 24004 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (6s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 01:57:58 (1788587878) striped dir -i0 -c2 -H all_char /mnt/lustre/d14ab.sanityn open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 24595 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (6s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 01:58:04 (1788587884) striped dir -i0 -c2 -H crush2 /mnt/lustre/d14b.sanityn truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 25185 Terminated $DIR1/$tdir/sleep 60 PASS 14b (6s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 01:58:11 (1788587891) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d14c.sanityn cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 25780 Terminated $DIR1/$tdir/sleep 60 PASS 14c (6s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 01:58:17 (1788587897) striped dir -i0 -c2 -H crush /mnt/lustre/d14d.sanityn chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 26375 Terminated $DIR1/$tdir/sleep 60 PASS 14d (8s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 01:58:25 (1788587905) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7530496 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7530496KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (51s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 01:59:16 (1788587956) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.716576 s, 11.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.4358 s, 19.2 MB/s Chance of close/open is 1 in 50 Seed set to 7961 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[0] 1788587962.907193 mapwrite 0x7b7e5a thru 0x7bcfbd (0x5164 bytes) 000003[0] 1788587963.173756 punch from 0x6e955d to 0x6f6b33 (0xd5d7 bytes) 000007[0] 1788587963.281324 punch from 0x255f35 to 0x2585fc (0x26c8 bytes) 000013[1] 1788587963.370445 read 0x6a58dc thru 0x6b482e (0xef53 bytes) 000017[0] 1788587963.470269 write 0x3f2c0c thru 0x3f6edd (0x42d2 bytes) 000018[0] 1788587963.490539 mapwrite 0x8ebbcd thru 0x8f4345 (0x8779 bytes) 000020[0] 1788587963.720356 read 0x51b2ce thru 0x51e3bc (0x30ef bytes) 000021[1] 1788587963.755103 punch from 0x5effeb to 0x5f53e3 (0x53f9 bytes) 000022[0] 1788587963.794230 mapread 0x6ba15f thru 0x6c3f54 (0x9df6 bytes) 000024[0] 1788587963.867396 write 0x288562 thru 0x293e3e (0xb8dd bytes) truncating to largest ever: 0xcabfd 000026[1] 1788587963.925682 trunc from 0x8f4346 to 0x0cabfc (0x829749 bytes) 000028[0] 1788587963.999170 mapread 0x03d015 thru 0x04bc05 (0xebf1 bytes) 000031[0] 1788587964.126653 mapwrite 0x657cb8 thru 0x65c99d (0x4ce6 bytes) 000033[1] 1788587964.323318 mapread 0x562cf3 thru 0x564267 (0x1575 bytes) truncating to largest ever: 0x21aac7 000034[0] 1788587964.378695 trunc from 0x65c99e to 0x21aac6 (0x441ed7 bytes) 000039[0] 1788587964.478466 punch from 0x1ed860 to 0x1f8314 (0xaab5 bytes) 000042[0] 1788587964.492728 punch from 0x0fdc1c to 0x0feb78 (0xf5d bytes) 000043[0] 1788587964.516240 mapwrite 0x6a024b thru 0x6aba40 (0xb7f6 bytes) 000044[1] 1788587964.726148 read 0x3f1690 thru 0x3f43b0 (0x2d21 bytes) 000049[0] 1788587964.832741 mapread 0x2e72c0 thru 0x2f55ba (0xe2fb bytes) 000053[0] 1788587964.932751 punch from 0x17157c to 0x17c630 (0xb0b5 bytes) truncating to largest ever: 0x92ea98 000054[0] 1788587964.975666 trunc from 0x6aba41 to 0x92ea97 (0x283057 bytes) 000055[1] 1788587964.997936 mapread 0x6b692a thru 0x6c37be (0xce95 bytes) 000056[0] 1788587965.073935 write 0x95e845 thru 0x964460 (0x5c1c bytes) 000057[0] 1788587965.081653 write 0x5fcc06 thru 0x606fc7 (0xa3c2 bytes) 000059[0] 1788587965.133942 mapread 0x230258 thru 0x23e22f (0xdfd8 bytes) 000061[1] 1788587965.208408 mapwrite 0x46f9d2 thru 0x4776fe (0x7d2d bytes) 000062[0] 1788587965.389444 mapwrite 0x038488 thru 0x044fde (0xcb57 bytes) 000063[1] 1788587965.527110 read 0x263c7b thru 0x2698d5 (0x5c5b bytes) 000064[1] 1788587965.571705 trunc from 0x964461 to 0x1954b1 (0x7cefaf bytes) 000066[1] 1788587965.694560 read 0x02350d thru 0x024462 (0xf56 bytes) 000069[0] 1788587965.785937 mapread 0x0deb09 thru 0x0ea417 (0xb90f bytes) 000070[0] 1788587965.869221 mapread 0x0a71f2 thru 0x0b47d9 (0xd5e8 bytes) 000071[1] 1788587965.903561 punch from 0x083c57 to 0x08c3ea (0x8794 bytes) 000073[1] 1788587965.937731 write 0x4b1e41 thru 0x4b9d8d (0x7f4d bytes) 000074[0] 1788587966.000699 mapwrite 0x739bc7 thru 0x73d565 (0x399f bytes) 000075[1] 1788587966.176705 trunc from 0x73d566 to 0x2c7ff0 (0x475575 bytes) 000077[1] 1788587966.292048 write 0x578115 thru 0x5782a5 (0x191 bytes) 000078[0] 1788587966.328759 write 0x5b71d8 thru 0x5bbb54 (0x497d bytes) 000081[0] 1788587966.442921 trunc from 0x5bbb55 to 0x049041 (0x572b13 bytes) 000082[0] 1788587966.477245 trunc from 0x049042 to 0x2144dc (0x1cb49b bytes) 000083[1] 1788587966.494704 trunc from 0x2144dd to 0x69bebf (0x4879e3 bytes) 000084[1] 1788587966.588772 read 0x3008b7 thru 0x309bfe (0x9348 bytes) 000085[0] 1788587966.602906 mapread 0x3ba351 thru 0x3c8402 (0xe0b2 bytes) 000086[1] 1788587966.670725 mapread 0x4706b0 thru 0x476e1f (0x6770 bytes) 000087[1] 1788587966.705828 trunc from 0x69bec0 to 0x558aeb (0x1433d4 bytes) 000091[1] 1788587966.820320 punch from 0x2aa1cb to 0x2b068c (0x64c2 bytes) 000092[1] 1788587966.840501 read 0x50a22b thru 0x513e45 (0x9c1b bytes) 000093[0] 1788587966.889432 mapread 0x2e7d6a thru 0x2f78ac (0xfb43 bytes) 000094[1] 1788587966.943220 punch from 0x28bf70 to 0x296bc0 (0xac51 bytes) 000097[1] 1788587966.979972 mapread 0x384512 thru 0x392a6f (0xe55e bytes) 000099[0] 1788587967.062737 write 0x4019c1 thru 0x40e076 (0xc6b6 bytes) 000100[0] 1788587967.111962 trunc from 0x558aec to 0x6e488b (0x18bda0 bytes) 000102[1] 1788587967.204659 write 0x928be5 thru 0x92bf0f (0x332b bytes) 000103[0] 1788587967.242419 mapread 0x4d718f thru 0x4e1a85 (0xa8f7 bytes) 000107[1] 1788587967.352143 trunc from 0x92bf10 to 0x8f2f42 (0x38fcd bytes) 000109[1] 1788587967.421338 punch from 0x350ce5 to 0x35c3d0 (0xb6ec bytes) 000111[1] 1788587967.465870 punch from 0x371907 to 0x3752e5 (0x39df bytes) 000114[1] 1788587967.518096 read 0x811293 thru 0x815fc5 (0x4d33 bytes) 000115[0] 1788587967.546263 write 0x7e621c thru 0x7e9128 (0x2f0d bytes) 000117[1] 1788587967.609939 mapwrite 0x1269bc thru 0x12ac5b (0x42a0 bytes) 000120[1] 1788587967.768687 punch from 0x5062db to 0x50c6c8 (0x63ee bytes) 000121[0] 1788587967.819468 mapread 0x3efc71 thru 0x3f7b6f (0x7eff bytes) 000123[0] 1788587967.909746 mapread 0x6de136 thru 0x6ec4fc (0xe3c7 bytes) 000124[0] 1788587967.992161 read 0x2a9843 thru 0x2aad60 (0x151e bytes) 000126[1] 1788587968.003774 write 0x670005 thru 0x67c0fc (0xc0f8 bytes) 000127[0] 1788587968.060060 punch from 0x7a6a14 to 0x7ab6fc (0x4ce9 bytes) 000131[0] 1788587968.137768 mapread 0x35349a thru 0x3591a2 (0x5d09 bytes) 000132[0] 1788587968.181279 trunc from 0x8f2f43 to 0x614e5c (0x2de0e6 bytes) 000133[0] 1788587968.279366 mapwrite 0x36bcae thru 0x379dc7 (0xe11a bytes) 000135[0] 1788587968.476829 write 0x09c8a1 thru 0x0a179b (0x4efb bytes) 000137[1] 1788587968.516068 trunc from 0x614e5d to 0x30f3d9 (0x305a83 bytes) 000138[1] 1788587968.615545 punch from 0x006563 to 0x00b235 (0x4cd3 bytes) 000139[1] 1788587968.645868 trunc from 0x30f3da to 0x371063 (0x61c8a bytes) 000140[1] 1788587968.677220 read 0x1cce7a thru 0x1d7702 (0xa889 bytes) 000141[1] 1788587968.770104 trunc from 0x371064 to 0x8752bf (0x50425c bytes) 000143[0] 1788587968.839031 write 0x4a461d thru 0x4b070e (0xc0f2 bytes) 000145[1] 1788587968.903631 read 0x0a8a85 thru 0x0b3262 (0xa7de bytes) 000146[0] 1788587968.917714 punch from 0x5b389c to 0x5c096f (0xd0d4 bytes) 000147[1] 1788587968.935264 trunc from 0x8752c0 to 0x37ffe5 (0x4f52da bytes) 000149[0] 1788587968.998894 trunc from 0x37ffe6 to 0x159930 (0x2266b5 bytes) 000151[0] 1788587969.054922 mapread 0x09a7c5 thru 0x0a1d64 (0x75a0 bytes) 000154[0] 1788587969.117665 mapread 0x073b80 thru 0x077c6a (0x40eb bytes) 000156[1] 1788587969.166937 punch from 0x14299e to 0x151b25 (0xf188 bytes) 000158[1] 1788587969.230405 punch from 0x0f4f55 to 0x0fe2a2 (0x934e bytes) 000159[0] 1788587969.256089 punch from 0x01ad14 to 0x01b6ce (0x9bb bytes) 000160[1] 1788587969.269075 mapread 0x0b600a thru 0x0b8935 (0x292c bytes) 000162[0] 1788587969.352206 mapwrite 0x38dc98 thru 0x38eb0b (0xe74 bytes) 000163[1] 1788587969.474910 write 0x814471 thru 0x8219aa (0xd53a bytes) 000164[0] 1788587969.542421 mapwrite 0x71baf6 thru 0x72ac68 (0xf173 bytes) 000169[1] 1788587970.191780 mapread 0x1213b8 thru 0x12bcee (0xa937 bytes) 000170[1] 1788587970.245322 read 0x49663b thru 0x4a4d2b (0xe6f1 bytes) 000171[0] 1788587970.289118 punch from 0x251d6f to 0x25b646 (0x98d8 bytes) 000173[1] 1788587970.375067 write 0x19cd11 thru 0x1a741e (0xa70e bytes) 000174[1] 1788587970.418103 mapwrite 0x6dd1a3 thru 0x6e6c9e (0x9afc bytes) 000175[0] 1788587970.527013 trunc from 0x8219ab to 0x6de047 (0x143963 bytes) 000177[1] 1788587970.613279 write 0x96ac94 thru 0x972af1 (0x7e5e bytes) 000178[1] 1788587970.687424 write 0x42447a thru 0x42b40d (0x6f94 bytes) 000180[1] 1788587970.823079 mapread 0x1e8ec9 thru 0x1eaa1f (0x1b57 bytes) 000181[1] 1788587970.897646 punch from 0x8f6873 to 0x901206 (0xa994 bytes) 000183[1] 1788587970.976487 punch from 0x1d13be to 0x1de631 (0xd274 bytes) 000187[0] 1788587971.091898 write 0x62a9e2 thru 0x6384d9 (0xdaf8 bytes) 000188[1] 1788587971.148333 mapwrite 0x49b480 thru 0x49cb14 (0x1695 bytes) 000190[1] 1788587971.273952 punch from 0x881038 to 0x88cafe (0xbac7 bytes) 000191[1] 1788587971.324673 read 0x71e664 thru 0x721d20 (0x36bd bytes) 000192[0] 1788587971.360114 write 0x4c05ed thru 0x4c1adc (0x14f0 bytes) 000195[0] 1788587971.480914 mapwrite 0x32fdca thru 0x33cff5 (0xd22c bytes) 000196[0] 1788587971.621703 mapread 0x7ad92a thru 0x7b4b2e (0x7205 bytes) 000197[1] 1788587971.668405 punch from 0x828cc2 to 0x83262a (0x9969 bytes) 000198[1] 1788587971.711765 punch from 0x011307 to 0x020e01 (0xfafb bytes) 000199[1] 1788587971.737916 mapread 0x5867ee thru 0x58c6a8 (0x5ebb bytes) 000200[0] 1788587971.861909 mapread 0x061661 thru 0x06cb90 (0xb530 bytes) 000201[0] 1788587971.947347 write 0x628df0 thru 0x62ac94 (0x1ea5 bytes) 000205[0] 1788587972.000638 read 0x33d366 thru 0x34a857 (0xd4f2 bytes) 000206[1] 1788587972.042731 read 0x23168e thru 0x23a48f (0x8e02 bytes) 000208[0] 1788587972.091629 read 0x63cd69 thru 0x647953 (0xabeb bytes) 000209[1] 1788587972.156698 trunc from 0x972af2 to 0x0c9686 (0x8a946b bytes) 000210[0] 1788587972.235597 write 0x1c166b thru 0x1c33ff (0x1d95 bytes) 000212[1] 1788587972.295469 trunc from 0x1c3400 to 0x0a2dd5 (0x12062a bytes) 000214[1] 1788587972.333911 trunc from 0x0a2dd6 to 0x7c54eb (0x722716 bytes) 000215[0] 1788587972.374326 trunc from 0x7c54ec to 0x76b719 (0x59dd2 bytes) 000216[0] 1788587972.475707 mapread 0x754908 thru 0x7560c9 (0x17c2 bytes) 000217[0] 1788587972.575723 write 0x754978 thru 0x7592a4 (0x492d bytes) 000221[0] 1788587972.728794 mapread 0x638572 thru 0x64592b (0xd3ba bytes) 000222[0] 1788587972.772289 write 0x0cab7c thru 0x0da4c6 (0xf94b bytes) 000223[0] 1788587972.807600 punch from 0x4c521f to 0x4d3f0f (0xecf1 bytes) 000224[0] 1788587972.818442 mapread 0x3ae4ec thru 0x3b8397 (0x9eac bytes) 000226[0] 1788587972.842606 write 0x3068bd thru 0x306902 (0x46 bytes) 000227[1] 1788587972.848465 read 0x34a9c0 thru 0x34e4aa (0x3aeb bytes) 000228[0] 1788587972.910158 trunc from 0x76b71a to 0x347e99 (0x423880 bytes) 000233[0] 1788587973.008767 write 0x86b50d thru 0x871e12 (0x6906 bytes) 000235[0] 1788587973.058121 mapwrite 0x81c3c5 thru 0x822658 (0x6294 bytes) 000236[0] 1788587973.141701 mapread 0x370448 thru 0x378805 (0x83be bytes) 000239[0] 1788587973.196305 mapread 0x74bb53 thru 0x75373f (0x7bed bytes) 000240[0] 1788587973.222564 write 0x49edd0 thru 0x4a11e3 (0x2414 bytes) 000243[0] 1788587973.260094 punch from 0x351609 to 0x360130 (0xeb28 bytes) 000244[0] 1788587973.272609 mapread 0x870265 thru 0x871e12 (0x1bae bytes) 000246[1] 1788587973.330990 mapwrite 0x5b3130 thru 0x5c13ea (0xe2bb bytes) 000247[0] 1788587974.033243 punch from 0x32cd35 to 0x331b80 (0x4e4c bytes) 000249[1] 1788587974.092579 punch from 0x3f9bc8 to 0x3fdf25 (0x435e bytes) 000251[1] 1788587974.139354 read 0x82ba2f thru 0x82e890 (0x2e62 bytes) 000252[0] 1788587974.165398 trunc from 0x871e13 to 0x206ba4 (0x66b26e bytes) 000253[0] 1788587974.211578 trunc from 0x206ba5 to 0x5dd146 (0x3d65a2 bytes) 000254[0] 1788587974.230540 write 0x027e3e thru 0x028c0a (0xdcd bytes) 000255[0] 1788587974.264105 punch from 0x5415d5 to 0x548c41 (0x766d bytes) 000256[0] 1788587974.269665 mapread 0x0f82d4 thru 0x100d95 (0x8ac2 bytes) 000257[1] 1788587974.288547 punch from 0x0bd9b1 to 0x0be2ea (0x93a bytes) 000261[0] 1788587974.385393 write 0x14708c thru 0x15267e (0xb5f3 bytes) 000263[1] 1788587974.482537 trunc from 0x5dd147 to 0x373180 (0x269fc6 bytes) 000265[1] 1788587974.589414 punch from 0x2fc5a7 to 0x3085c8 (0xc022 bytes) 000266[1] 1788587974.604461 write 0x7cbd2f thru 0x7d49c5 (0x8c97 bytes) 000267[0] 1788587974.609417 mapwrite 0x7cc075 thru 0x7d36e2 (0x766e bytes) 000268[0] 1788587974.720884 read 0x5372dd thru 0x540c18 (0x993c bytes) 000272[1] 1788587974.787107 read 0x13b532 thru 0x149f6c (0xea3b bytes) 000274[1] 1788587974.826185 read 0x3a1500 thru 0x3af654 (0xe155 bytes) 000275[1] 1788587974.911290 mapwrite 0x629abd thru 0x635366 (0xb8aa bytes) 000276[1] 1788587975.009909 mapwrite 0x8ad981 thru 0x8ae4c8 (0xb48 bytes) 000277[1] 1788587975.081329 mapread 0x1c8b9b thru 0x1ce78a (0x5bf0 bytes) 000279[1] 1788587975.135302 mapread 0x23acbf thru 0x23df16 (0x3258 bytes) 000281[1] 1788587975.194425 punch from 0x065369 to 0x06e07b (0x8d13 bytes) 000282[0] 1788587975.203170 punch from 0x4a2cd4 to 0x4ad2f6 (0xa623 bytes) 000284[1] 1788587975.256962 write 0x3d474a thru 0x3d7172 (0x2a29 bytes) 000285[0] 1788587975.284320 punch from 0x672ef1 to 0x67ed77 (0xbe87 bytes) 000287[0] 1788587975.323393 trunc from 0x8ae4c9 to 0x52c62a (0x381e9e bytes) 000289[1] 1788587975.431128 trunc from 0x52c62b to 0x4cf326 (0x5d304 bytes) 000292[1] 1788587975.544168 mapread 0x34d1a3 thru 0x351088 (0x3ee6 bytes) 000294[0] 1788587975.575252 mapwrite 0x864ddb thru 0x86580a (0xa30 bytes) 000295[0] 1788587975.751277 mapread 0x131f95 thru 0x13e162 (0xc1ce bytes) 000296[0] 1788587975.790352 trunc from 0x86580b to 0x725cc5 (0x13fb45 bytes) 000297[0] 1788587975.843767 punch from 0x082dd9 to 0x0885d2 (0x57fa bytes) 000298[1] 1788587975.852662 trunc from 0x725cc6 to 0x105cc5 (0x620000 bytes) 000299[1] 1788587975.931109 trunc from 0x105cc6 to 0x306668 (0x2009a3 bytes) 000300[1] 1788587975.985202 punch from 0x04d8ad to 0x04f51d (0x1c71 bytes) 000302[1] 1788587976.002748 mapread 0x12dcfe thru 0x1329f0 (0x4cf3 bytes) 000303[0] 1788587976.074744 read 0x2eedea thru 0x2fe2cd (0xf4e4 bytes) 000308[0] 1788587976.291254 mapread 0x2113b5 thru 0x21dc13 (0xc85f bytes) 000309[1] 1788587976.331632 mapread 0x0dce57 thru 0x0e9774 (0xc91e bytes) 000310[1] 1788587976.362472 trunc from 0x306669 to 0x6ae942 (0x3a82da bytes) 000311[0] 1788587976.432918 mapwrite 0x3986a9 thru 0x39fac4 (0x741c bytes) truncating to largest ever: 0x971e3a 000313[0] 1788587976.590095 trunc from 0x6ae943 to 0x971e39 (0x2c34f7 bytes) 000314[0] 1788587976.718251 read 0x54ed22 thru 0x550401 (0x16e0 bytes) 000315[0] 1788587976.742685 mapread 0x20b3b8 thru 0x2152b9 (0x9f02 bytes) 000316[1] 1788587976.798490 mapwrite 0x3c0f38 thru 0x3c51bb (0x4284 bytes) 000317[1] 1788587976.984286 read 0x3bf9a9 thru 0x3c5f20 (0x6578 bytes) 000318[0] 1788587976.997372 punch from 0x422230 to 0x42b59c (0x936d bytes) 000319[0] 1788587977.038709 mapread 0x8d9cfd thru 0x8df3fd (0x5701 bytes) 000320[1] 1788587977.131123 punch from 0x44fde8 to 0x459f54 (0xa16d bytes) 000321[0] 1788587977.154986 mapread 0x8a005a thru 0x8a4687 (0x462e bytes) 000322[1] 1788587977.179433 punch from 0x1b1c55 to 0x1b96f0 (0x7a9c bytes) 000323[1] 1788587977.197359 mapread 0x845895 thru 0x853624 (0xdd90 bytes) 000324[1] 1788587977.245500 punch from 0x314876 to 0x315878 (0x1003 bytes) 000325[1] 1788587977.250469 mapread 0x13d88b thru 0x14bb26 (0xe29c bytes) 000327[1] 1788587977.283501 write 0x6ff427 thru 0x709d3e (0xa918 bytes) 000328[0] 1788587977.306680 write 0x3cb64d thru 0x3d9f72 (0xe926 bytes) 000330[0] 1788587977.402791 trunc from 0x971e3a to 0x1c2c24 (0x7af215 bytes) 000331[0] 1788587977.482786 mapread 0x025757 thru 0x0265b9 (0xe63 bytes) 000335[0] 1788587977.528124 mapwrite 0x1b4bc8 thru 0x1bc7c3 (0x7bfc bytes) 000336[1] 1788587977.609849 write 0x285980 thru 0x28d757 (0x7dd8 bytes) 000341[0] 1788587977.795232 mapread 0x0a1b7d thru 0x0ac987 (0xae0b bytes) 000343[1] 1788587977.972840 trunc from 0x28d758 to 0x1edcfa (0x9fa5d bytes) 000344[0] 1788587978.077440 read 0x18097c thru 0x188c7a (0x82ff bytes) 000345[1] 1788587978.219049 write 0x2f5047 thru 0x2f8581 (0x353b bytes) 000347[0] 1788587978.260051 trunc from 0x2f8582 to 0x459988 (0x161407 bytes) 000349[0] 1788587978.361294 mapread 0x0ff0aa thru 0x103a16 (0x496d bytes) 000350[1] 1788587978.444966 write 0x021385 thru 0x023d2c (0x29a8 bytes) 000351[1] 1788587978.481433 punch from 0x002382 to 0x00dce4 (0xb963 bytes) 000352[1] 1788587978.521755 trunc from 0x459989 to 0x7ccd4e (0x3733c6 bytes) 000353[1] 1788587978.587065 punch from 0x54068d to 0x545a71 (0x53e5 bytes) 000356[0] 1788587978.668281 mapread 0x3d7a44 thru 0x3e3583 (0xbb40 bytes) 000359[0] 1788587978.853707 trunc from 0x7ccd4f to 0x01c804 (0x7b054a bytes) 000360[0] 1788587978.902824 punch from 0x014583 to 0x014891 (0x30f bytes) 000361[1] 1788587978.908315 punch from 0x008d2d to 0x015c2b (0xceff bytes) 000362[0] 1788587978.937613 mapread 0x0102da thru 0x01c804 (0xc52b bytes) 000363[1] 1788587979.059876 punch from 0x01bf38 to 0x01c804 (0x8cd bytes) 000365[0] 1788587979.158731 trunc from 0x01c805 to 0x56c32a (0x54fb26 bytes) 000366[1] 1788587979.263118 write 0x604ce2 thru 0x611783 (0xcaa2 bytes) 000370[0] 1788587979.376765 read 0x36e255 thru 0x37b9de (0xd78a bytes) 000372[0] 1788587979.432307 punch from 0x174982 to 0x17ecb8 (0xa337 bytes) 000373[1] 1788587979.476230 punch from 0x24673a to 0x24a1a7 (0x3a6e bytes) 000375[0] 1788587979.509310 mapwrite 0x84f8fc thru 0x85da7e (0xe183 bytes) 000376[1] 1788587979.867708 trunc from 0x85da7f to 0x44c12f (0x41194f bytes) 000378[1] 1788587979.914333 read 0x31f847 thru 0x31f892 (0x4c bytes) 000381[1] 1788587979.923674 mapread 0x205897 thru 0x20de82 (0x85ec bytes) 000382[0] 1788587979.948565 read 0x07c1a9 thru 0x081c29 (0x5a81 bytes) 000383[1] 1788587980.031528 punch from 0x37e7b9 to 0x383cd9 (0x5521 bytes) 000384[0] 1788587980.078664 read 0x37fab6 thru 0x38e942 (0xee8d bytes) 000385[0] 1788587980.140348 write 0x34f52f thru 0x350bcb (0x169d bytes) 000386[0] 1788587980.204695 mapread 0x0b412c thru 0x0c3392 (0xf267 bytes) 000388[0] 1788587980.264920 mapwrite 0x203b0d thru 0x204670 (0xb64 bytes) 000390[0] 1788587980.413263 mapread 0x25cfc7 thru 0x266bd9 (0x9c13 bytes) 000391[1] 1788587980.471113 mapwrite 0x7e3986 thru 0x7ee37b (0xa9f6 bytes) 000392[1] 1788587980.690715 punch from 0x6095dd to 0x60d817 (0x423b bytes) 000393[0] 1788587980.700891 mapwrite 0x3abdde thru 0x3b0b6b (0x4d8e bytes) 000395[1] 1788587980.888890 punch from 0x2ee9d8 to 0x2fbd3b (0xd364 bytes) 000399[1] 1788587980.978734 trunc from 0x7ee37c to 0x8c0778 (0xd23fd bytes) 000400[0] 1788587981.015453 mapwrite 0x7a2644 thru 0x7a2c23 (0x5e0 bytes) 000402[0] 1788587981.197132 trunc from 0x8c0779 to 0x623dfd (0x29c97b bytes) 000403[1] 1788587981.252192 punch from 0x3b9b22 to 0x3c9038 (0xf517 bytes) 000405[0] 1788587981.334403 trunc from 0x623dfe to 0x31bdbe (0x30803f bytes) 000407[0] 1788587981.402252 mapwrite 0x03c849 thru 0x03d56e (0xd26 bytes) 000411[0] 1788587981.519799 write 0x51015f thru 0x516488 (0x632a bytes) 000412[1] 1788587981.522760 write 0x83eaee thru 0x83ecc6 (0x1d9 bytes) 000413[1] 1788587981.553777 trunc from 0x83ecc7 to 0x64eb90 (0x1f0136 bytes) 000414[1] 1788587981.610004 read 0x35a54b thru 0x36782e (0xd2e4 bytes) 000417[1] 1788587981.820571 punch from 0x347e34 to 0x34b346 (0x3513 bytes) 000420[0] 1788587981.925743 mapread 0x1d5544 thru 0x1d9eba (0x4977 bytes) 000423[0] 1788587982.060591 mapwrite 0x68023a thru 0x689f46 (0x9d0d bytes) 000424[1] 1788587982.292149 mapwrite 0x823c9b thru 0x829874 (0x5bda bytes) 000425[0] 1788587982.441477 mapwrite 0x3ff58c thru 0x409517 (0x9f8c bytes) 000426[1] 1788587982.551588 punch from 0x187d96 to 0x18b182 (0x33ed bytes) 000430[1] 1788587982.706639 trunc from 0x829875 to 0x774dee (0xb4a86 bytes) 000432[0] 1788587982.825646 write 0x14e81e thru 0x156105 (0x78e8 bytes) 000433[1] 1788587982.916504 read 0x2788e8 thru 0x280fc5 (0x86de bytes) 000434[1] 1788587982.978175 write 0x20a138 thru 0x20b9b1 (0x187a bytes) 000436[0] 1788587983.093630 read 0x3545df thru 0x3645b4 (0xffd6 bytes) 000438[0] 1788587983.237036 punch from 0x4a98fb to 0x4ace96 (0x359c bytes) 000441[1] 1788587983.345895 punch from 0x482089 to 0x48cf04 (0xae7c bytes) 000442[0] 1788587983.386099 mapread 0x3de928 thru 0x3ecdd1 (0xe4aa bytes) 000443[0] 1788587983.420546 read 0x2fadfe thru 0x2fba5a (0xc5d bytes) 000444[0] 1788587983.521458 mapread 0x1eafba thru 0x1f9e59 (0xeea0 bytes) 000446[0] 1788587983.566918 mapwrite 0x0d4981 thru 0x0d7e36 (0x34b6 bytes) 000447[1] 1788587983.827279 mapread 0x02d82a thru 0x02e593 (0xd6a bytes) 000448[1] 1788587983.902997 mapwrite 0x37df1c thru 0x37ebae (0xc93 bytes) 000451[0] 1788587984.105323 mapwrite 0x2d886c thru 0x2d8fee (0x783 bytes) 000452[1] 1788587984.282689 trunc from 0x774def to 0x8a82a1 (0x1334b3 bytes) 000453[0] 1788587984.357460 punch from 0x12039b to 0x124230 (0x3e96 bytes) 000454[1] 1788587984.404302 punch from 0x26e2fd to 0x275637 (0x733b bytes) 000456[1] 1788587984.485212 punch from 0x02ff0e to 0x03398e (0x3a81 bytes) 000458[1] 1788587984.496785 punch from 0x1bcc38 to 0x1bfbff (0x2fc8 bytes) 000460[0] 1788587984.553456 mapread 0x11c499 thru 0x1299bc (0xd524 bytes) 000462[1] 1788587984.732536 read 0x3d1d45 thru 0x3d5ee1 (0x419d bytes) 000463[1] 1788587984.748669 mapread 0x1f78a7 thru 0x1f8bd2 (0x132c bytes) 000464[1] 1788587984.766760 mapwrite 0x0c9898 thru 0x0cb302 (0x1a6b bytes) 000466[1] 1788587984.985707 mapwrite 0x7b2a7a thru 0x7b2acb (0x52 bytes) 000468[1] 1788587985.297670 write 0x3d2508 thru 0x3d6db0 (0x48a9 bytes) 000469[0] 1788587985.349856 punch from 0x7c6032 to 0x7cba91 (0x5a60 bytes) 000470[0] 1788587985.387378 read 0x5a624a thru 0x5a80a6 (0x1e5d bytes) 000475[1] 1788587985.520733 trunc from 0x8a82a2 to 0x3ecac8 (0x4bb7d9 bytes) 000477[0] 1788587985.633632 write 0x99df96 thru 0x9aaee4 (0xcf4f bytes) 000479[0] 1788587985.826160 trunc from 0x9aaee5 to 0x6262be (0x384c26 bytes) 000480[0] 1788587985.982311 mapread 0x35f0b1 thru 0x36c72a (0xd67a bytes) 000481[0] 1788587986.032657 read 0x461037 thru 0x46dc37 (0xcc01 bytes) 000483[0] 1788587986.132443 mapwrite 0x1997d5 thru 0x19e01a (0x4846 bytes) 000484[1] 1788587986.307043 mapwrite 0x8ecb50 thru 0x8f25f9 (0x5aaa bytes) 000485[0] 1788587986.431202 read 0x8a9ea1 thru 0x8ad156 (0x32b6 bytes) 000486[1] 1788587986.465803 read 0x7497ee thru 0x74e4c9 (0x4cdc bytes) 000487[1] 1788587986.494598 read 0x77b229 thru 0x789646 (0xe41e bytes) truncating to largest ever: 0x9bd8c0 000489[0] 1788587986.509974 trunc from 0x8f25fa to 0x9bd8bf (0xcb2c6 bytes) 000490[0] 1788587986.688079 punch from 0x3ec732 to 0x3ef9ce (0x329d bytes) 000493[0] 1788587986.788442 mapread 0x16979a thru 0x169cfd (0x564 bytes) 000496[0] 1788587986.884023 punch from 0x7499d8 to 0x74d549 (0x3b72 bytes) 000499[0] 1788587986.956850 read 0x346174 thru 0x354b94 (0xea21 bytes) 000500[1] 1788587986.968322 trunc from 0x9bd8c0 to 0x297914 (0x725fab bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 7987 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003[1] 1788587987.911870 write_OD 0x3c5000 thru 0x3cefff (0xa000 bytes) 000005[1] 1788587988.100083 mapread 0x2c9000 thru 0x2cdfde (0x4fdf bytes) 000007[0] 1788587988.130018 mapread 0x139000 thru 0x13de81 (0x4e82 bytes) 000008[0] 1788587988.150132 punch from 0x013653 to 0x01f26c (0xbc1a bytes) 000010[1] 1788587988.196726 punch from 0x37b755 to 0x38437b (0x8c27 bytes) 000011[0] 1788587988.271619 read 0x38f000 thru 0x38f2d7 (0x2d8 bytes) truncating to largest ever: 0x8c0bcd 000015[0] 1788587988.320151 trunc from 0x3cf000 to 0x8c0bcc (0x4f1bcd bytes) 000016[1] 1788587988.391457 mapwrite 0x663000 thru 0x6668c1 (0x38c2 bytes) 000017[1] 1788587988.507073 write_OD 0x84f000 thru 0x85bfff (0xd000 bytes) 000018[0] 1788587988.569494 punch from 0x1df329 to 0x1e60e0 (0x6db8 bytes) 000019[1] 1788587988.614161 punch from 0x7ce86c to 0x7d89de (0xa173 bytes) 000020[0] 1788587988.659091 write 0x2b0000 thru 0x2b7595 (0x7596 bytes) 000021[0] 1788587988.690980 punch from 0x673480 to 0x674e5f (0x19e0 bytes) 000022[0] 1788587988.697939 read 0x244000 thru 0x251ce0 (0xdce1 bytes) 000026[0] 1788587988.763426 mapread 0x2ef000 thru 0x2f9585 (0xa586 bytes) 000027[0] 1788587988.812793 write 0x2ea000 thru 0x2f86c6 (0xe6c7 bytes) 000029[0] 1788587988.827583 read 0x674000 thru 0x67734d (0x334e bytes) 000031[1] 1788587988.832538 mapread 0x2ea000 thru 0x2f6f73 (0xcf74 bytes) 000032[1] 1788587988.887274 punch from 0x1294af to 0x133b68 (0xa6ba bytes) 000034[0] 1788587988.917129 read 0x717000 thru 0x726f4e (0xff4f bytes) 000035[0] 1788587988.946761 write 0x73d000 thru 0x74a928 (0xd929 bytes) 000038[1] 1788587989.011603 read_OD 0x777000 thru 0x77afff (0x4000 bytes) 000039[0] 1788587989.062780 write 0x36a000 thru 0x3752c7 (0xb2c8 bytes) 000040[0] 1788587989.133040 mapread 0x88a000 thru 0x88e2fb (0x42fc bytes) 000041[0] 1788587989.233565 trunc from 0x8c0bcd to 0x162e41 (0x75dd8b bytes) 000043[0] 1788587989.327268 write 0x9c2000 thru 0x9d0e2e (0xee2f bytes) 000044[1] 1788587989.354444 mapread 0x70e000 thru 0x70fd4a (0x1d4b bytes) 000045[0] 1788587989.466258 mapread 0x41c000 thru 0x41cd82 (0xd83 bytes) 000046[0] 1788587989.493911 punch from 0x2f3fa2 to 0x2f5074 (0x10d3 bytes) 000047[1] 1788587989.560471 mapwrite 0x184000 thru 0x1874a0 (0x34a1 bytes) 000049[0] 1788587989.721480 write 0x9c8000 thru 0x9c98fa (0x18fb bytes) 000050[1] 1788587989.780357 write_OD 0x2c2000 thru 0x2cefff (0xd000 bytes) 000051[0] 1788587990.003075 mapwrite 0x1a2000 thru 0x1a85b1 (0x65b2 bytes) 000052[0] 1788587990.208738 write 0x93d000 thru 0x946a22 (0x9a23 bytes) 000053[0] 1788587990.230465 mapwrite 0x811000 thru 0x81893c (0x793d bytes) 000057[1] 1788587990.482252 mapread 0x02e000 thru 0x036a94 (0x8a95 bytes) 000058[0] 1788587990.564377 trunc from 0x9d0e2f to 0x434ffb (0x59be33 bytes) 000059[1] 1788587990.605433 write_OD 0x512000 thru 0x519fff (0x8000 bytes) 000060[0] 1788587990.683863 trunc from 0x51a000 to 0x86af8a (0x350f8b bytes) 000062[1] 1788587990.742369 read_OD 0x757000 thru 0x761fff (0xb000 bytes) 000064[1] 1788587990.781098 write_OD 0x1bd000 thru 0x1c0fff (0x4000 bytes) 000065[1] 1788587990.873076 trunc from 0x86af8b to 0x2646b0 (0x6068da bytes) 000066[0] 1788587990.936349 mapwrite 0x19b000 thru 0x1a612f (0xb130 bytes) truncating to largest ever: 0x9bcb1a 000067[0] 1788587991.062070 trunc from 0x2646b1 to 0x9bcb19 (0x758469 bytes) 000068[0] 1788587991.126437 mapread 0x0cf000 thru 0x0d7b80 (0x8b81 bytes) 000069[0] 1788587991.139581 read 0x082000 thru 0x085181 (0x3182 bytes) 000070[0] 1788587991.144801 write 0x245000 thru 0x2487d0 (0x37d1 bytes) 000071[0] 1788587991.150800 mapwrite 0x788000 thru 0x78e62c (0x662d bytes) 000072[1] 1788587991.243258 trunc from 0x9bcb1a to 0x45959c (0x56357d bytes) 000077[1] 1788587991.324035 mapread 0x3fa000 thru 0x406af4 (0xcaf5 bytes) 000078[1] 1788587991.370080 mapread 0x01e000 thru 0x02c63b (0xe63c bytes) 000079[1] 1788587991.443406 write_OD 0x983000 thru 0x98efff (0xc000 bytes) 000083[1] 1788587991.593557 trunc from 0x98f000 to 0x4fd0b7 (0x491f48 bytes) 000084[1] 1788587991.714148 trunc from 0x4fd0b8 to 0x0753db (0x487cdc bytes) 000085[0] 1788587991.819619 read 0x028000 thru 0x02a893 (0x2894 bytes) 000086[1] 1788587991.865361 punch from 0x037d59 to 0x038d02 (0xfaa bytes) 000089[0] 1788587991.942604 punch from 0x0696ce to 0x074770 (0xb0a3 bytes) 000090[1] 1788587991.957674 mapwrite 0x175000 thru 0x184b9e (0xfb9f bytes) 000092[1] 1788587992.235565 write_OD 0x41e000 thru 0x422fff (0x5000 bytes) 000093[0] 1788587992.349089 write 0x9fd000 thru 0x9ff9be (0x29bf bytes) 000094[0] 1788587992.388893 trunc from 0x9ff9bf to 0x8829ee (0x17cfd0 bytes) 000095[1] 1788587992.451096 trunc from 0x8829ef to 0x597a34 (0x2eafba bytes) 000096[0] 1788587992.515210 trunc from 0x597a35 to 0x9ba22d (0x4227f9 bytes) 000099[0] 1788587992.711094 mapwrite 0x130000 thru 0x13e066 (0xe067 bytes) 000100[0] 1788587992.898302 mapread 0x772000 thru 0x77ea84 (0xca85 bytes) 000102[0] 1788587993.009974 read 0x67d000 thru 0x6839f8 (0x69f9 bytes) 000104[0] 1788587993.045762 write 0x507000 thru 0x50db23 (0x6b24 bytes) 000105[1] 1788587993.062661 mapread 0x071000 thru 0x07ca43 (0xba44 bytes) 000106[1] 1788587993.145507 trunc from 0x9ba22e to 0x7d82c3 (0x1e1f6a bytes) 000107[0] 1788587993.201762 read 0x382000 thru 0x38bf46 (0x9f47 bytes) 000108[0] 1788587993.225600 mapwrite 0x69e000 thru 0x69e26b (0x26c bytes) 000112[1] 1788587993.355045 trunc from 0x7d82c4 to 0x6e4bd1 (0xf36f2 bytes) 000114[0] 1788587993.424722 punch from 0x60d986 to 0x6146f6 (0x6d71 bytes) 000116[1] 1788587993.459122 write_OD 0x410000 thru 0x410fff (0x1000 bytes) 000117[0] 1788587993.524454 write 0x21a000 thru 0x224d97 (0xad98 bytes) 000118[1] 1788587993.561019 punch from 0x4ce4bb to 0x4d76c4 (0x920a bytes) 000122[1] 1788587993.701208 punch from 0x58b426 to 0x59abf3 (0xf7ce bytes) 000123[1] 1788587993.746839 punch from 0x571de1 to 0x5770ca (0x52ea bytes) 000125[0] 1788587993.869300 read 0x62e000 thru 0x62fc8e (0x1c8f bytes) 000126[1] 1788587993.944105 mapwrite 0x7a6000 thru 0x7b185d (0xb85e bytes) 000128[1] 1788587994.353929 read_OD 0x1aa000 thru 0x1b6fff (0xd000 bytes) 000129[0] 1788587994.415466 punch from 0x06e109 to 0x06f3ea (0x12e2 bytes) 000134[0] 1788587994.597410 write 0x2d7000 thru 0x2dc88a (0x588b bytes) 000135[1] 1788587994.633432 mapwrite 0x6cb000 thru 0x6d7345 (0xc346 bytes) 000137[1] 1788587994.980227 mapwrite 0x2a5000 thru 0x2affa3 (0xafa4 bytes) 000140[1] 1788587995.219050 mapread 0x0b9000 thru 0x0b9a37 (0xa38 bytes) 000141[1] 1788587995.243605 mapwrite 0x7d3000 thru 0x7e15b3 (0xe5b4 bytes) 000142[0] 1788587995.530914 mapwrite 0x6e5000 thru 0x6e5731 (0x732 bytes) 000143[1] 1788587995.703013 punch from 0x0c968a to 0x0d5b9b (0xc512 bytes) 000144[0] 1788587995.751723 punch from 0x46bb05 to 0x471ad1 (0x5fcd bytes) 000145[0] 1788587995.781126 mapread 0x282000 thru 0x289b22 (0x7b23 bytes) 000148[1] 1788587995.857636 punch from 0x2700bb to 0x27f02c (0xef72 bytes) 000153[1] 1788587995.919380 write_OD 0x45b000 thru 0x465fff (0xb000 bytes) 000155[1] 1788587996.079081 mapwrite 0x39a000 thru 0x39d1fc (0x31fd bytes) 000156[1] 1788587996.255725 read_OD 0x331000 thru 0x33cfff (0xc000 bytes) 000157[1] 1788587996.310963 punch from 0x65b4b2 to 0x65d03c (0x1b8b bytes) 000161[1] 1788587996.441535 mapread 0x0ac000 thru 0x0ba68a (0xe68b bytes) 000162[1] 1788587996.559850 trunc from 0x7e15b4 to 0x903141 (0x121b8e bytes) 000163[0] 1788587996.623347 read 0x432000 thru 0x438ffd (0x6ffe bytes) 000168[1] 1788587996.681639 trunc from 0x903142 to 0x07a804 (0x88893d bytes) 000170[0] 1788587996.776979 read 0x05e000 thru 0x06bbe2 (0xdbe3 bytes) 000171[0] 1788587996.804373 read 0x055000 thru 0x05ef0e (0x9f0f bytes) 000173[0] 1788587996.839067 trunc from 0x07a805 to 0x5cf1a0 (0x55499c bytes) 000175[0] 1788587996.949040 trunc from 0x5cf1a1 to 0x3afde1 (0x21f3bf bytes) 000176[1] 1788587997.013059 mapwrite 0x902000 thru 0x9042d3 (0x22d4 bytes) 000180[0] 1788587997.192602 read 0x2d0000 thru 0x2d85e6 (0x85e7 bytes) 000181[0] 1788587997.237940 write 0x3c7000 thru 0x3d06c2 (0x96c3 bytes) 000183[1] 1788587997.343540 mapread 0x08f000 thru 0x09e0f4 (0xf0f5 bytes) 000184[1] 1788587997.442769 mapread 0x5dc000 thru 0x5e81f3 (0xc1f4 bytes) 000185[0] 1788587997.521709 mapread 0x3b5000 thru 0x3bbbab (0x6bac bytes) 000186[0] 1788587997.557540 mapread 0x45b000 thru 0x45b64c (0x64d bytes) 000188[1] 1788587997.569554 mapwrite 0x3a3000 thru 0x3abfd5 (0x8fd6 bytes) 000189[0] 1788587997.724156 mapwrite 0x438000 thru 0x43c5ed (0x45ee bytes) 000192[1] 1788587997.871210 mapread 0x484000 thru 0x48ef38 (0xaf39 bytes) 000193[0] 1788587997.919537 mapwrite 0x64a000 thru 0x64ef04 (0x4f05 bytes) 000196[1] 1788587998.071588 punch from 0x4a7728 to 0x4b4c1f (0xd4f8 bytes) 000197[1] 1788587998.112651 read_OD 0x74d000 thru 0x756fff (0xa000 bytes) 000198[1] 1788587998.141487 write_OD 0x13a000 thru 0x13bfff (0x2000 bytes) 000199[1] 1788587998.217791 mapwrite 0x9e7000 thru 0x9ea20e (0x320f bytes) 000201[0] 1788587998.369510 read 0x12d000 thru 0x130c79 (0x3c7a bytes) 000202[0] 1788587998.420130 trunc from 0x9ea20f to 0x8d999a (0x110874 bytes) 000203[1] 1788587998.495942 write_OD 0x3a0000 thru 0x3a4fff (0x5000 bytes) 000206[1] 1788587998.658765 read_OD 0x11e000 thru 0x128fff (0xb000 bytes) 000207[0] 1788587998.688696 punch from 0x5f9eac to 0x608c04 (0xed59 bytes) 000208[0] 1788587998.742335 write 0x90b000 thru 0x90e0c0 (0x30c1 bytes) 000209[0] 1788587998.743613 write 0x422000 thru 0x424b6f (0x2b70 bytes) 000210[0] 1788587998.756293 trunc from 0x90e0c1 to 0x432a63 (0x4db65d bytes) 000211[0] 1788587998.825695 mapread 0x0e0000 thru 0x0e28ec (0x28ed bytes) 000212[0] 1788587998.907155 write 0x781000 thru 0x784cdb (0x3cdc bytes) 000213[1] 1788587998.931490 write_OD 0x627000 thru 0x62dfff (0x7000 bytes) 000214[0] 1788587999.027349 read 0x29f000 thru 0x2a696c (0x796d bytes) 000215[1] 1788587999.062278 trunc from 0x784cdc to 0x61c553 (0x168788 bytes) 000218[1] 1788587999.175639 write_OD 0x1c0000 thru 0x1c9fff (0xa000 bytes) 000221[1] 1788587999.314628 mapwrite 0x2c8000 thru 0x2d2435 (0xa436 bytes) 000222[0] 1788587999.544603 trunc from 0x61c554 to 0x914eb5 (0x2f8962 bytes) 000224[1] 1788587999.647676 write_OD 0x71d000 thru 0x724fff (0x8000 bytes) 000226[1] 1788587999.712573 mapwrite 0x6f1000 thru 0x6f6085 (0x5086 bytes) 000227[1] 1788587999.810575 trunc from 0x914eb6 to 0x414324 (0x500b91 bytes) 000228[1] 1788587999.851528 trunc from 0x414325 to 0x12c2ec (0x2e8038 bytes) 000229[1] 1788587999.882777 mapwrite 0x609000 thru 0x6115f0 (0x85f1 bytes) 000232[1] 1788588000.014691 read_OD 0x4f6000 thru 0x503fff (0xe000 bytes) 000237[1] 1788588000.080499 punch from 0x1369be to 0x13d466 (0x6aa9 bytes) 000240[1] 1788588000.118060 mapwrite 0x516000 thru 0x51e8e6 (0x88e7 bytes) 000241[1] 1788588000.211084 trunc from 0x6115f1 to 0x3a656c (0x26b084 bytes) 000242[1] 1788588000.232061 trunc from 0x3a656d to 0x1adca1 (0x1f88cb bytes) 000243[1] 1788588000.271640 mapwrite 0x53a000 thru 0x53ed7e (0x4d7f bytes) 000244[1] 1788588000.546005 trunc from 0x53ed7f to 0x0408e3 (0x4fe49b bytes) 000245[0] 1788588000.637988 read 0x03d000 thru 0x03fe8f (0x2e90 bytes) 000249[1] 1788588000.779285 mapread 0x003000 thru 0x00d18c (0xa18d bytes) 000251[1] 1788588000.815030 mapwrite 0x9b1000 thru 0x9bace8 (0x9ce9 bytes) 000253[1] 1788588000.986494 write_OD 0x63d000 thru 0x63ffff (0x3000 bytes) 000256[0] 1788588001.077185 trunc from 0x9bace9 to 0x8693a6 (0x151942 bytes) 000258[0] 1788588001.149515 mapread 0x1c9000 thru 0x1d4bf9 (0xbbfa bytes) 000260[0] 1788588001.307238 write 0x188000 thru 0x196090 (0xe091 bytes) 000261[0] 1788588001.330158 write 0x469000 thru 0x4712db (0x82dc bytes) 000263[0] 1788588001.386761 write 0x107000 thru 0x1076e0 (0x6e1 bytes) 000264[0] 1788588001.440273 trunc from 0x8693a7 to 0x5dc278 (0x28d12e bytes) 000265[1] 1788588001.483680 mapread 0x320000 thru 0x32e840 (0xe841 bytes) 000266[1] 1788588001.711162 mapwrite 0x59c000 thru 0x59f446 (0x3447 bytes) 000267[0] 1788588001.883184 mapwrite 0x569000 thru 0x57780a (0xe80b bytes) 000269[1] 1788588002.147098 mapread 0x27d000 thru 0x281000 (0x4001 bytes) 000270[1] 1788588002.246799 trunc from 0x5dc279 to 0x603ea2 (0x27c2a bytes) 000272[1] 1788588002.371194 mapwrite 0x1e2000 thru 0x1eee92 (0xce93 bytes) 000273[0] 1788588002.618375 punch from 0x5e55f9 to 0x5ef674 (0xa07c bytes) 000275[1] 1788588002.688787 mapwrite 0x5b2000 thru 0x5bf80f (0xd810 bytes) truncating to largest ever: 0x9d6348 000276[0] 1788588002.936038 trunc from 0x603ea3 to 0x9d6347 (0x3d24a5 bytes) 000281[0] 1788588003.047242 write 0x0d0000 thru 0x0db0e0 (0xb0e1 bytes) 000284[0] 1788588003.054401 punch from 0x751f3b to 0x75a9c3 (0x8a89 bytes) 000286[1] 1788588003.107304 mapwrite 0x5aa000 thru 0x5aad7f (0xd80 bytes) truncating to largest ever: 0x9f866a 000287[1] 1788588003.360541 trunc from 0x9d6348 to 0x9f8669 (0x22322 bytes) 000289[0] 1788588003.403041 punch from 0x7bbf23 to 0x7cbd52 (0xfe30 bytes) 000290[1] 1788588003.425990 mapwrite 0x75b000 thru 0x765b00 (0xab01 bytes) 000291[0] 1788588003.545579 read 0x5b1000 thru 0x5bed9d (0xdd9e bytes) 000294[0] 1788588003.581344 trunc from 0x9f866a to 0x41aee8 (0x5dd781 bytes) 000298[1] 1788588003.675159 read_OD 0x0e3000 thru 0x0edfff (0xb000 bytes) 000299[1] 1788588003.716170 mapread 0x38a000 thru 0x394ffb (0xaffc bytes) 000300[0] 1788588003.743425 punch from 0x3c00aa to 0x3c62fb (0x6252 bytes) 000302[1] 1788588003.800702 trunc from 0x41aee9 to 0x1e5e9f (0x235049 bytes) 000303[1] 1788588003.847636 punch from 0x0756d4 to 0x084a06 (0xf333 bytes) 000305[0] 1788588003.889682 write 0x6a7000 thru 0x6a97d0 (0x27d1 bytes) 000307[1] 1788588003.974514 write_OD 0x0c0000 thru 0x0c0fff (0x1000 bytes) 000308[0] 1788588004.037109 mapread 0x616000 thru 0x618b01 (0x2b02 bytes) 000309[0] 1788588004.060435 mapwrite 0x60b000 thru 0x60e6dc (0x36dd bytes) 000310[0] 1788588004.223500 trunc from 0x6a97d1 to 0x7b05a5 (0x106dd5 bytes) 000313[0] 1788588004.362406 mapread 0x6fb000 thru 0x6fd819 (0x281a bytes) 000315[0] 1788588004.381512 mapwrite 0x431000 thru 0x431840 (0x841 bytes) 000316[1] 1788588004.561933 trunc from 0x7b05a6 to 0x7a7afe (0x8aa7 bytes) 000318[1] 1788588004.687477 trunc from 0x7a7aff to 0x4aa0d8 (0x2fda26 bytes) 000319[1] 1788588004.721494 mapwrite 0x281000 thru 0x2906f2 (0xf6f3 bytes) 000320[0] 1788588004.965998 read 0x386000 thru 0x38eb06 (0x8b07 bytes) 000322[1] 1788588005.035351 trunc from 0x4aa0d9 to 0x0e238d (0x3c7d4b bytes) 000323[1] 1788588005.165101 read_OD 0x00d000 thru 0x00efff (0x2000 bytes) 000324[1] 1788588005.226303 punch from 0x06de7d to 0x074284 (0x6408 bytes) 000325[0] 1788588005.271105 trunc from 0x0e238e to 0x27c856 (0x19a4c9 bytes) 000329[1] 1788588005.427068 write_OD 0x834000 thru 0x839fff (0x6000 bytes) 000332[1] 1788588005.544531 write_OD 0x221000 thru 0x227fff (0x7000 bytes) 000333[1] 1788588005.613475 mapread 0x586000 thru 0x58edc2 (0x8dc3 bytes) 000334[0] 1788588005.850271 mapread 0x812000 thru 0x8154b2 (0x34b3 bytes) 000335[0] 1788588005.916318 mapread 0x206000 thru 0x20cf6d (0x6f6e bytes) 000336[1] 1788588005.963308 read_OD 0x09c000 thru 0x0a6fff (0xb000 bytes) 000338[1] 1788588006.011286 read_OD 0x610000 thru 0x61cfff (0xd000 bytes) 000339[0] 1788588006.051330 trunc from 0x83a000 to 0x5c79b8 (0x272647 bytes) 000342[0] 1788588006.183676 punch from 0x0ce0db to 0x0da120 (0xc046 bytes) 000343[1] 1788588006.214894 punch from 0x5b4e46 to 0x5bb3d0 (0x658b bytes) 000346[0] 1788588006.302649 write 0x399000 thru 0x39f9d6 (0x69d7 bytes) 000347[1] 1788588006.344315 trunc from 0x5c79b9 to 0x972c7d (0x3ab2c5 bytes) 000348[1] 1788588006.488499 mapread 0x614000 thru 0x616c51 (0x2c52 bytes) 000349[1] 1788588006.532353 punch from 0x692562 to 0x69ccd2 (0xa771 bytes) 000350[1] 1788588006.562224 trunc from 0x972c7e to 0x13192b (0x841352 bytes) 000351[1] 1788588006.586664 mapread 0x03e000 thru 0x0430ea (0x50eb bytes) 000352[0] 1788588006.618608 mapwrite 0x645000 thru 0x64b47b (0x647c bytes) 000353[1] 1788588006.763573 mapread 0x04a000 thru 0x0507ff (0x6800 bytes) 000355[1] 1788588006.817386 punch from 0x252a4d to 0x258e98 (0x644c bytes) 000356[1] 1788588006.838140 trunc from 0x64b47c to 0x135f27 (0x515554 bytes) 000359[1] 1788588006.878675 mapwrite 0x9bf000 thru 0x9c0074 (0x1075 bytes) 000360[1] 1788588007.057185 trunc from 0x9c0075 to 0x48a4d6 (0x535b9e bytes) 000361[1] 1788588007.174546 punch from 0x224fe5 to 0x22b206 (0x6222 bytes) 000364[1] 1788588007.215391 read_OD 0x414000 thru 0x418fff (0x5000 bytes) 000365[0] 1788588007.265732 trunc from 0x48a4d7 to 0x459573 (0x30f63 bytes) 000366[0] 1788588007.365271 read 0x140000 thru 0x141212 (0x1213 bytes) 000367[0] 1788588007.386650 mapread 0x270000 thru 0x27bc23 (0xbc24 bytes) 000369[1] 1788588007.446112 trunc from 0x459574 to 0x302837 (0x156d3c bytes) 000370[0] 1788588007.531108 trunc from 0x302838 to 0x3626d0 (0x5fe99 bytes) 000379[0] 1788588007.723291 mapread 0x212000 thru 0x2185ef (0x65f0 bytes) 000380[1] 1788588007.759199 punch from 0x24e9e6 to 0x25c4ca (0xdae5 bytes) 000382[1] 1788588007.782580 punch from 0x0dcace to 0x0e2c8b (0x61be bytes) 000383[0] 1788588007.823793 mapwrite 0x848000 thru 0x848ce7 (0xce8 bytes) 000384[0] 1788588008.046541 mapwrite 0x9f7000 thru 0x9fa86d (0x386e bytes) 000385[1] 1788588008.323166 mapwrite 0x3d6000 thru 0x3df7cf (0x97d0 bytes) 000386[0] 1788588008.537893 mapwrite 0x3b8000 thru 0x3bd6e4 (0x56e5 bytes) 000387[0] 1788588008.777909 mapwrite 0x4d8000 thru 0x4df541 (0x7542 bytes) 000388[0] 1788588008.881312 write 0x7b2000 thru 0x7bdbce (0xbbcf bytes) 000389[1] 1788588008.904305 mapread 0x0a0000 thru 0x0a6500 (0x6501 bytes) 000391[0] 1788588009.045544 trunc from 0x9fa86e to 0x57f31f (0x47b54e bytes) 000393[0] 1788588009.141585 punch from 0x0a1d01 to 0x0a62a1 (0x45a1 bytes) 000400[0] 1788588009.271895 mapread 0x0e3000 thru 0x0e90ac (0x60ad bytes) 000401[0] 1788588009.416680 read 0x2eb000 thru 0x2ec9dd (0x19de bytes) 000402[0] 1788588009.439868 write 0x8d3000 thru 0x8dea62 (0xba63 bytes) 000403[0] 1788588009.445395 mapwrite 0x96e000 thru 0x976454 (0x8455 bytes) 000406[0] 1788588009.785517 trunc from 0x976455 to 0x1fa63c (0x77be18 bytes) 000407[1] 1788588009.828332 mapwrite 0x4fd000 thru 0x50bb78 (0xeb79 bytes) 000409[1] 1788588010.048132 mapread 0x259000 thru 0x260192 (0x7193 bytes) 000412[0] 1788588010.145284 trunc from 0x50bb79 to 0x529d20 (0x1e1a8 bytes) 000413[1] 1788588010.214100 mapwrite 0x5c1000 thru 0x5ca835 (0x9836 bytes) 000414[1] 1788588010.479548 punch from 0x3e6de3 to 0x3e740d (0x62b bytes) 000416[0] 1788588010.524293 mapwrite 0x771000 thru 0x777933 (0x6934 bytes) 000417[1] 1788588010.722284 punch from 0x35121a to 0x3572ce (0x60b5 bytes) 000418[1] 1788588010.749265 trunc from 0x777934 to 0x7334c4 (0x4446f bytes) 000419[1] 1788588010.778348 read 0x4d0000 thru 0x4d6fec (0x6fed bytes) 000420[0] 1788588010.800993 read 0x2ff000 thru 0x305b11 (0x6b12 bytes) 000421[1] 1788588010.844436 read 0x3aa000 thru 0x3acf0c (0x2f0d bytes) 000423[0] 1788588010.858980 mapwrite 0x96a000 thru 0x974bb5 (0xabb6 bytes) 000424[1] 1788588011.092276 mapread 0x6c1000 thru 0x6c71d9 (0x61da bytes) 000425[1] 1788588011.203610 trunc from 0x974bb6 to 0x79e638 (0x1d657d bytes) 000427[1] 1788588011.391162 mapwrite 0x66e000 thru 0x66ef0d (0xf0e bytes) 000428[0] 1788588011.510634 mapread 0x23e000 thru 0x24a8e6 (0xc8e7 bytes) 000429[1] 1788588011.577529 read 0x561000 thru 0x566b21 (0x5b22 bytes) 000430[1] 1788588011.588922 trunc from 0x79e639 to 0x8e6e00 (0x1487c8 bytes) 000431[0] 1788588011.632699 trunc from 0x8e6e01 to 0x6acf05 (0x239efb bytes) 000433[0] 1788588011.787907 read 0x287000 thru 0x28bc6f (0x4c70 bytes) 000436[1] 1788588011.893542 mapread 0x19d000 thru 0x1a3128 (0x6129 bytes) 000437[1] 1788588012.025532 trunc from 0x6acf06 to 0x29b780 (0x411785 bytes) 000438[0] 1788588012.107578 write 0x47e000 thru 0x4822fc (0x42fd bytes) 000439[0] 1788588012.115947 read 0x11e000 thru 0x121438 (0x3439 bytes) 000442[0] 1788588012.203464 mapwrite 0x6b3000 thru 0x6bb600 (0x8601 bytes) 000443[0] 1788588012.442117 trunc from 0x6bb601 to 0x834032 (0x178a32 bytes) 000444[1] 1788588012.476578 read 0x21a000 thru 0x21bf82 (0x1f83 bytes) 000446[1] 1788588012.525752 read 0x109000 thru 0x1173a2 (0xe3a3 bytes) 000448[0] 1788588012.558672 read 0x301000 thru 0x304d67 (0x3d68 bytes) 000452[0] 1788588012.568476 read 0x276000 thru 0x27ebc5 (0x8bc6 bytes) 000453[0] 1788588012.585368 mapwrite 0x7d7000 thru 0x7d83f4 (0x13f5 bytes) 000454[0] 1788588012.690852 punch from 0x4a1a0c to 0x4a34bf (0x1ab4 bytes) 000461[1] 1788588012.858789 punch from 0x14f1e8 to 0x15cbab (0xd9c4 bytes) 000464[1] 1788588012.904551 trunc from 0x834033 to 0x0d72dd (0x75cd55 bytes) 000465[0] 1788588012.959932 read 0x070000 thru 0x076d2e (0x6d2f bytes) 000466[0] 1788588013.069442 write 0x39f000 thru 0x3a97b3 (0xa7b4 bytes) 000469[1] 1788588013.125982 trunc from 0x3a97b4 to 0x3bf586 (0x15dd3 bytes) 000470[0] 1788588013.214112 punch from 0x1cfa33 to 0x1ddfe3 (0xe5b1 bytes) 000471[0] 1788588013.285349 punch from 0x369013 to 0x37384f (0xa83d bytes) 000472[1] 1788588013.297684 mapwrite 0x49e000 thru 0x49f8b5 (0x18b6 bytes) 000473[1] 1788588013.435323 mapwrite 0x3c4000 thru 0x3c6e2c (0x2e2d bytes) 000474[0] 1788588013.572439 write 0x7f7000 thru 0x8069d7 (0xf9d8 bytes) 000476[1] 1788588013.666764 mapwrite 0x273000 thru 0x27fb12 (0xcb13 bytes) 000477[0] 1788588013.939836 mapwrite 0x76f000 thru 0x77cbb5 (0xdbb6 bytes) 000479[1] 1788588014.143177 trunc from 0x8069d8 to 0x690225 (0x1767b2 bytes) 000481[0] 1788588014.235532 write 0x0e0000 thru 0x0eeb3a (0xeb3b bytes) 000482[0] 1788588014.265255 write 0x853000 thru 0x856edb (0x3edc bytes) 000483[1] 1788588014.295140 punch from 0x010bfe to 0x017fc6 (0x73c9 bytes) 000486[1] 1788588014.386709 mapread 0x561000 thru 0x56dc3a (0xcc3b bytes) 000487[0] 1788588014.461077 read 0x43d000 thru 0x446305 (0x9306 bytes) 000488[0] 1788588014.494251 read 0x594000 thru 0x596681 (0x2682 bytes) 000489[0] 1788588014.514131 trunc from 0x856edc to 0x666d4c (0x1f018f bytes) 000491[1] 1788588014.584107 read 0x174000 thru 0x17d755 (0x9756 bytes) 000492[1] 1788588014.614098 trunc from 0x666d4d to 0x8a4c0d (0x23dec1 bytes) 000493[0] 1788588014.682394 write 0x209000 thru 0x21545b (0xc45c bytes) 000494[0] 1788588014.728526 read 0x79d000 thru 0x7ac315 (0xf316 bytes) 000495[0] 1788588014.744406 read 0x360000 thru 0x36dd40 (0xdd41 bytes) 000497[1] 1788588014.758637 read 0x1d6000 thru 0x1dc319 (0x631a bytes) 000498[1] 1788588014.801511 write 0x163000 thru 0x165ac0 (0x2ac1 bytes) 000500[1] 1788588014.882517 punch from 0x1a5a06 to 0x1ace2c (0x7427 bytes) All operations completed A-OK! PASS 16a (65s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 02:00:21 (1788588021) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.521695 s, 16.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.336 s, 25.0 MB/s Chance of close/open is 1 in 50 Seed set to 8025 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole 000004[0] 1788588026.278280 mapwrite 0x78faec thru 0x790e0a (0x131f bytes) 000005[0] 1788588026.467967 punch from 0x12d49b to 0x12dcd8 (0x83e bytes) 000006[0] 1788588026.509727 punch from 0x5a1d30 to 0x5a3345 (0x1616 bytes) truncating to largest ever: 0x79585e 000007[1] 1788588026.533043 trunc from 0x790e0b to 0x79585d (0x4a53 bytes) 000008[0] 1788588026.601047 mapwrite 0x5cdb2c thru 0x5cfad6 (0x1fab bytes) 000009[0] 1788588026.734478 write 0x630de1 thru 0x631446 (0x666 bytes) 000010[1] 1788588026.781839 mapwrite 0x32a51f thru 0x32c22d (0x1d0f bytes) 000013[1] 1788588026.977733 mapwrite 0x70720b thru 0x7088f5 (0x16eb bytes) 000014[0] 1788588027.120322 punch from 0x061f87 to 0x0636e8 (0x1762 bytes) 000016[1] 1788588027.211518 punch from 0x4ef19d to 0x4ef74c (0x5b0 bytes) 000019[1] 1788588027.307578 mapwrite 0x6b8c1e thru 0x6b8ca4 (0x87 bytes) 000023[1] 1788588027.435788 read 0x589575 thru 0x58af3d (0x19c9 bytes) 000025[1] 1788588027.474362 punch from 0x25806a to 0x258a16 (0x9ad bytes) 000026[0] 1788588027.510781 write 0x522ef8 thru 0x5231b2 (0x2bb bytes) 000027[1] 1788588027.564539 trunc from 0x79585e to 0x1d778b (0x5be0d2 bytes) 000028[0] 1788588027.678875 write 0x371624 thru 0x372042 (0xa1f bytes) 000031[1] 1788588027.745142 mapread 0x0c86c5 thru 0x0c8f3d (0x879 bytes) truncating to largest ever: 0x83c044 000032[1] 1788588027.801459 trunc from 0x372043 to 0x83c043 (0x4ca001 bytes) 000033[0] 1788588027.839667 trunc from 0x83c044 to 0x0455d7 (0x7f6a6c bytes) 000034[1] 1788588027.887614 mapread 0x033fc5 thru 0x035eaf (0x1eeb bytes) 000035[0] 1788588027.940169 mapread 0x01ddb3 thru 0x01e06d (0x2bb bytes) 000036[0] 1788588027.970617 punch from 0x030ef1 to 0x030f45 (0x55 bytes) 000039[1] 1788588028.051674 write 0x2217b5 thru 0x22285b (0x10a7 bytes) 000041[0] 1788588028.095448 mapread 0x0c5aa6 thru 0x0c6493 (0x9ee bytes) 000042[0] 1788588028.135386 punch from 0x172b11 to 0x174967 (0x1e57 bytes) 000043[1] 1788588028.158087 write 0x133a18 thru 0x134be2 (0x11cb bytes) 000045[0] 1788588028.260282 mapread 0x03324a thru 0x033740 (0x4f7 bytes) 000046[0] 1788588028.364641 read 0x0e51e3 thru 0x0e6b99 (0x19b7 bytes) 000047[1] 1788588028.371947 write 0x5ad3e5 thru 0x5ae5f1 (0x120d bytes) 000048[0] 1788588028.392995 mapread 0x32281d thru 0x323a46 (0x122a bytes) 000049[1] 1788588028.452057 punch from 0x135ff3 to 0x13723e (0x124c bytes) 000050[0] 1788588028.522180 write 0x895108 thru 0x895576 (0x46f bytes) 000056[0] 1788588028.648946 punch from 0x8558ba to 0x856048 (0x78f bytes) 000058[0] 1788588028.711802 mapwrite 0x21ad45 thru 0x21bfe2 (0x129e bytes) 000059[1] 1788588028.831094 write 0x81cf02 thru 0x81d321 (0x420 bytes) 000060[1] 1788588028.870285 mapwrite 0x58157c thru 0x582290 (0xd15 bytes) 000062[1] 1788588029.021995 read 0x86526f thru 0x8664de (0x1270 bytes) 000063[0] 1788588029.033349 mapwrite 0x2690ff thru 0x26961d (0x51f bytes) 000064[1] 1788588029.138346 mapread 0x627ae2 thru 0x628552 (0xa71 bytes) 000065[0] 1788588029.184196 trunc from 0x895577 to 0x7255dc (0x16ff9a bytes) 000071[1] 1788588029.334066 punch from 0x55a964 to 0x55b483 (0xb20 bytes) 000073[0] 1788588029.390850 read 0x013f3a thru 0x0152fd (0x13c4 bytes) 000076[1] 1788588029.444094 write 0x412a56 thru 0x412d29 (0x2d4 bytes) 000080[0] 1788588029.567287 trunc from 0x7255dd to 0x82155a (0xfbf7e bytes) 000082[0] 1788588029.701115 punch from 0x0d1f72 to 0x0d3895 (0x1924 bytes) 000083[1] 1788588029.744908 write 0x3fe7c2 thru 0x3ffd9c (0x15db bytes) 000084[1] 1788588029.801804 mapwrite 0x510360 thru 0x5103c2 (0x63 bytes) 000086[0] 1788588029.925211 read 0x4353c2 thru 0x43568f (0x2ce bytes) truncating to largest ever: 0x8cf773 000090[1] 1788588029.974991 trunc from 0x82155b to 0x8cf772 (0xae218 bytes) 000091[0] 1788588030.013261 punch from 0x491c25 to 0x49316d (0x1549 bytes) 000092[1] 1788588030.045281 mapwrite 0x34d39b thru 0x34d8a8 (0x50e bytes) 000095[0] 1788588030.148925 write 0x938cc5 thru 0x93a6c8 (0x1a04 bytes) 000096[0] 1788588030.161514 read 0x255428 thru 0x25545b (0x34 bytes) 000097[0] 1788588030.193566 read 0x84959d thru 0x84af58 (0x19bc bytes) 000100[0] 1788588030.263129 trunc from 0x93a6c9 to 0x0cdf8f (0x86c739 bytes) 000102[1] 1788588030.343035 mapread 0x080b35 thru 0x081201 (0x6cd bytes) 000104[0] 1788588030.457219 read 0x09c5e6 thru 0x09ceb0 (0x8cb bytes) 000107[0] 1788588030.567472 read 0x04c9a4 thru 0x04d278 (0x8d5 bytes) 000109[0] 1788588030.621701 read 0x02f1e3 thru 0x03027a (0x1098 bytes) 000110[0] 1788588030.622563 trunc from 0x0cdf90 to 0x41677c (0x3487ed bytes) 000112[0] 1788588030.754688 punch from 0x2bd16f to 0x2be229 (0x10bb bytes) 000114[1] 1788588030.784501 read 0x1c0682 thru 0x1c0d81 (0x700 bytes) 000115[1] 1788588030.818330 read 0x38e978 thru 0x38ea63 (0xec bytes) 000116[1] 1788588030.849068 mapread 0x1356f1 thru 0x13622f (0xb3f bytes) 000118[1] 1788588030.894487 write 0x91d3ed thru 0x91eecc (0x1ae0 bytes) 000119[1] 1788588030.933565 mapwrite 0x0bcc38 thru 0x0bdc29 (0xff2 bytes) 000120[0] 1788588031.011307 write 0x53b79d thru 0x53c189 (0x9ed bytes) 000121[0] 1788588031.032110 trunc from 0x91eecd to 0x3058b0 (0x61961c bytes) 000122[0] 1788588031.096014 trunc from 0x3058b1 to 0x86c70c (0x566e5c bytes) 000124[0] 1788588031.171600 trunc from 0x86c70d to 0x31c8b7 (0x54fe55 bytes) 000125[1] 1788588031.212431 punch from 0x29be65 to 0x29bed8 (0x74 bytes) 000126[0] 1788588031.254575 mapread 0x1f3755 thru 0x1f4d7e (0x162a bytes) 000128[0] 1788588031.359271 write 0x3e8cdd thru 0x3e915c (0x480 bytes) 000130[0] 1788588031.412763 read 0x15e46a thru 0x15f3fc (0xf93 bytes) 000131[1] 1788588031.512714 punch from 0x16d604 to 0x16e0c1 (0xabe bytes) 000133[1] 1788588031.590750 write 0x9089a3 thru 0x90958d (0xbeb bytes) 000135[0] 1788588031.680212 mapwrite 0x338765 thru 0x33961a (0xeb6 bytes) 000137[0] 1788588031.839652 read 0x0bc7c9 thru 0x0be2ca (0x1b02 bytes) 000138[1] 1788588031.851239 mapread 0x2a7c35 thru 0x2a9989 (0x1d55 bytes) 000139[1] 1788588031.942945 trunc from 0x90958e to 0x02b6d5 (0x8ddeb8 bytes) 000140[1] 1788588032.052341 mapwrite 0x81ca36 thru 0x81ea11 (0x1fdc bytes) 000141[1] 1788588032.181630 write 0x65ad14 thru 0x65bae7 (0xdd4 bytes) 000144[0] 1788588032.246016 mapwrite 0x3dd4b0 thru 0x3decc9 (0x181a bytes) 000145[0] 1788588032.335566 read 0x6944cc thru 0x695100 (0xc35 bytes) 000147[1] 1788588032.391623 write 0x77d962 thru 0x77f2bf (0x195e bytes) 000148[1] 1788588032.454290 trunc from 0x81ea12 to 0x48636c (0x3986a5 bytes) 000149[1] 1788588032.531244 read 0x04d463 thru 0x04da44 (0x5e2 bytes) 000151[0] 1788588032.544796 write 0x4548d6 thru 0x454fd3 (0x6fe bytes) 000153[1] 1788588032.602566 trunc from 0x48636d to 0x27e203 (0x208169 bytes) 000156[1] 1788588032.685370 mapwrite 0x4699e9 thru 0x469e8e (0x4a6 bytes) 000159[1] 1788588032.819925 write 0x56602d thru 0x5663e7 (0x3bb bytes) 000160[0] 1788588032.844012 trunc from 0x5663e8 to 0x7a4f9c (0x23ebb5 bytes) 000161[0] 1788588032.914469 read 0x0251e3 thru 0x025a19 (0x837 bytes) 000163[0] 1788588032.947316 mapread 0x45e71d thru 0x45f7b5 (0x1099 bytes) 000164[0] 1788588032.973445 trunc from 0x7a4f9d to 0x7fee42 (0x59ea6 bytes) 000165[1] 1788588032.993935 trunc from 0x7fee43 to 0x31421b (0x4eac27 bytes) 000166[0] 1788588033.038435 mapread 0x2a4e77 thru 0x2a5bee (0xd78 bytes) 000169[1] 1788588033.121180 mapwrite 0x055313 thru 0x0568a5 (0x1593 bytes) 000170[1] 1788588033.222403 mapread 0x0588ed thru 0x0598e9 (0xffd bytes) 000171[1] 1788588033.270412 mapread 0x0fcb29 thru 0x0fce6a (0x342 bytes) 000172[0] 1788588033.317066 write 0x16aa12 thru 0x16c0ee (0x16dd bytes) 000173[1] 1788588033.357049 mapwrite 0x8687d5 thru 0x868c8a (0x4b6 bytes) 000174[1] 1788588033.492618 mapwrite 0x6f373b thru 0x6f51b5 (0x1a7b bytes) 000175[0] 1788588033.620762 mapwrite 0x65da98 thru 0x65debb (0x424 bytes) 000176[0] 1788588033.780920 write 0x469487 thru 0x46aee6 (0x1a60 bytes) 000178[0] 1788588033.866132 mapread 0x6d7874 thru 0x6d7e27 (0x5b4 bytes) 000180[0] 1788588033.940472 mapwrite 0x1a185f thru 0x1a1a9f (0x241 bytes) truncating to largest ever: 0x9e9256 000181[0] 1788588034.040543 trunc from 0x868c8b to 0x9e9255 (0x1805cb bytes) 000182[0] 1788588034.091337 mapread 0x0ed828 thru 0x0edbae (0x387 bytes) 000183[1] 1788588034.113475 punch from 0x2511cb to 0x252f92 (0x1dc8 bytes) 000185[0] 1788588034.181168 mapwrite 0x1b4b15 thru 0x1b5c43 (0x112f bytes) 000190[0] 1788588034.320732 mapwrite 0x73be5d thru 0x73cb51 (0xcf5 bytes) 000192[1] 1788588034.456499 read 0x38ae5a thru 0x38cdf6 (0x1f9d bytes) 000195[1] 1788588034.508992 trunc from 0x9e9256 to 0x8be3d2 (0x12ae83 bytes) 000199[0] 1788588034.848946 write 0x608c2f thru 0x60a840 (0x1c12 bytes) 000200[1] 1788588034.902877 read 0x240e2c thru 0x241911 (0xae6 bytes) 000202[1] 1788588034.961113 mapread 0x2896c8 thru 0x2898ec (0x225 bytes) 000204[1] 1788588035.002616 punch from 0x69296c to 0x693bf8 (0x128d bytes) 000205[1] 1788588035.023485 trunc from 0x8be3d3 to 0x0af99f (0x80ea33 bytes) 000208[0] 1788588035.075132 read 0x026531 thru 0x027141 (0xc11 bytes) 000209[0] 1788588035.153478 punch from 0x0637e7 to 0x063aac (0x2c6 bytes) 000210[1] 1788588035.193581 mapwrite 0x0406fd thru 0x040cef (0x5f3 bytes) 000211[1] 1788588035.351378 mapread 0x0acffe thru 0x0ada48 (0xa4b bytes) 000212[0] 1788588035.452094 read 0x07a0c0 thru 0x07b366 (0x12a7 bytes) 000213[0] 1788588035.555985 punch from 0x034420 to 0x034f05 (0xae6 bytes) 000215[0] 1788588035.608957 trunc from 0x0af9a0 to 0x55487d (0x4a4ede bytes) 000217[1] 1788588035.638510 trunc from 0x55487e to 0x1e6e4d (0x36da30 bytes) 000218[0] 1788588035.684376 read 0x158721 thru 0x15a43f (0x1d1f bytes) 000219[1] 1788588035.722288 trunc from 0x1e6e4e to 0x205111 (0x1e2c4 bytes) 000220[0] 1788588035.756583 trunc from 0x205112 to 0x96d800 (0x7686ef bytes) 000222[1] 1788588035.829621 mapwrite 0x72750d thru 0x72830f (0xe03 bytes) 000223[1] 1788588035.927355 trunc from 0x96d801 to 0x2a067a (0x6cd186 bytes) 000225[1] 1788588036.010742 mapread 0x185ca4 thru 0x186dea (0x1147 bytes) 000227[1] 1788588036.026108 mapwrite 0x2a6b27 thru 0x2a8a44 (0x1f1e bytes) 000230[0] 1788588036.185552 trunc from 0x2a8a45 to 0x2109e9 (0x9805b bytes) 000233[0] 1788588036.274624 mapwrite 0x896b3a thru 0x898331 (0x17f8 bytes) 000235[1] 1788588036.454991 punch from 0x030ec4 to 0x032a5a (0x1b97 bytes) 000236[0] 1788588036.515685 punch from 0x1aa298 to 0x1ab401 (0x116a bytes) 000238[0] 1788588036.564375 mapwrite 0x03e89a thru 0x03f8fe (0x1065 bytes) 000240[0] 1788588036.654003 punch from 0x50723b to 0x5073da (0x1a0 bytes) 000241[1] 1788588036.692777 punch from 0x807c29 to 0x8093a8 (0x1780 bytes) 000242[1] 1788588036.722046 mapwrite 0x226c56 thru 0x228368 (0x1713 bytes) 000245[1] 1788588036.818798 write 0x9d4a90 thru 0x9d5d90 (0x1301 bytes) 000246[0] 1788588036.836113 write 0x28fb1f thru 0x291a67 (0x1f49 bytes) 000248[0] 1788588036.909605 write 0x57b551 thru 0x57bbf2 (0x6a2 bytes) 000249[1] 1788588036.916548 trunc from 0x9d5d91 to 0x968788 (0x6d608 bytes) 000250[0] 1788588037.011744 read 0x2d9fde thru 0x2da60c (0x62f bytes) 000251[1] 1788588037.040775 punch from 0x835648 to 0x835cd1 (0x68a bytes) 000252[1] 1788588037.055698 mapread 0x59bbd5 thru 0x59d55d (0x1989 bytes) 000253[0] 1788588037.108565 mapwrite 0x1e2ab4 thru 0x1e3826 (0xd73 bytes) 000255[0] 1788588037.208607 write 0x3251b4 thru 0x326e19 (0x1c66 bytes) 000256[1] 1788588037.237532 mapread 0x799537 thru 0x799679 (0x143 bytes) 000257[0] 1788588037.263232 trunc from 0x968789 to 0x3221f8 (0x646590 bytes) 000258[1] 1788588037.319085 write 0x82c7f8 thru 0x82e520 (0x1d29 bytes) 000260[1] 1788588037.378722 mapread 0x6b02dd thru 0x6b1818 (0x153c bytes) 000261[1] 1788588037.431451 read 0x65f071 thru 0x660976 (0x1906 bytes) 000262[0] 1788588037.438625 mapread 0x74a788 thru 0x74ae44 (0x6bd bytes) 000263[1] 1788588037.464123 read 0x1d32dc thru 0x1d4f91 (0x1cb6 bytes) 000264[1] 1788588037.475266 punch from 0x273260 to 0x275151 (0x1ef2 bytes) 000265[1] 1788588037.488261 trunc from 0x82e521 to 0x957fef (0x129acf bytes) 000267[0] 1788588037.566913 read 0x5e3f34 thru 0x5e5608 (0x16d5 bytes) 000269[1] 1788588037.604227 mapwrite 0x12697f thru 0x127b75 (0x11f7 bytes) 000270[1] 1788588037.691019 mapread 0x5ff5ab thru 0x5ff9f5 (0x44b bytes) 000272[1] 1788588037.741001 trunc from 0x957ff0 to 0x2a801a (0x6affd5 bytes) 000274[1] 1788588037.831314 read 0x13392f thru 0x13451c (0xbee bytes) 000276[1] 1788588037.962522 mapwrite 0x8f083b thru 0x8f0df8 (0x5be bytes) 000278[0] 1788588038.173532 mapwrite 0x4e14db thru 0x4e18dd (0x403 bytes) 000280[1] 1788588038.336879 mapread 0x15894b thru 0x158f71 (0x627 bytes) 000281[0] 1788588038.369949 mapread 0x3e102a thru 0x3e2663 (0x163a bytes) 000283[1] 1788588038.444048 mapwrite 0x5892bc thru 0x589c81 (0x9c6 bytes) 000284[1] 1788588038.522242 mapwrite 0x32d0d2 thru 0x32e780 (0x16af bytes) 000285[1] 1788588038.626126 trunc from 0x8f0df9 to 0x41a8f7 (0x4d6501 bytes) 000286[1] 1788588038.658297 mapread 0x0d7956 thru 0x0d9583 (0x1c2e bytes) 000287[1] 1788588038.706561 mapwrite 0x6933ed thru 0x694f99 (0x1bad bytes) 000289[0] 1788588038.816951 trunc from 0x694f9a to 0x8bc282 (0x2272e9 bytes) 000291[0] 1788588038.889732 trunc from 0x8bc283 to 0x33b9b0 (0x5808d2 bytes) 000294[0] 1788588038.947571 mapread 0x31ed78 thru 0x320b92 (0x1e1b bytes) 000296[0] 1788588038.986880 trunc from 0x33b9b1 to 0x99ee04 (0x663454 bytes) 000297[1] 1788588039.057417 mapread 0x89b59d thru 0x89bf4a (0x9ae bytes) 000300[1] 1788588039.156001 read 0x5abeae thru 0x5ad587 (0x16da bytes) 000303[0] 1788588039.194306 punch from 0x5d3e31 to 0x5d4393 (0x563 bytes) 000306[0] 1788588039.251377 trunc from 0x99ee05 to 0x7de421 (0x1c09e3 bytes) 000308[0] 1788588039.315188 mapread 0x285ce6 thru 0x286da0 (0x10bb bytes) 000309[1] 1788588039.326269 punch from 0x5f3c99 to 0x5f4fc0 (0x1328 bytes) 000310[1] 1788588039.361902 mapwrite 0x0d9f2a thru 0x0db142 (0x1219 bytes) 000311[0] 1788588039.457343 mapread 0x70210c thru 0x703b08 (0x19fd bytes) 000312[1] 1788588039.534180 punch from 0x0fe576 to 0x100169 (0x1bf4 bytes) 000314[1] 1788588039.540512 write 0x72db25 thru 0x72dddd (0x2b9 bytes) 000315[0] 1788588039.569291 mapwrite 0x620f25 thru 0x6211e5 (0x2c1 bytes) 000316[1] 1788588039.701872 write 0x102547 thru 0x1040e7 (0x1ba1 bytes) 000317[1] 1788588039.739688 write 0x0b3304 thru 0x0b3dc4 (0xac1 bytes) 000318[1] 1788588039.771665 mapread 0x4d4eea thru 0x4d61dd (0x12f4 bytes) 000320[1] 1788588039.812863 mapwrite 0x7e2980 thru 0x7e2de8 (0x469 bytes) 000321[0] 1788588039.983005 read 0x13c37d thru 0x13c43a (0xbe bytes) 000322[1] 1788588040.060220 read 0x4f2563 thru 0x4f31f1 (0xc8f bytes) 000324[1] 1788588040.135204 mapread 0x7c5767 thru 0x7c6edc (0x1776 bytes) 000326[0] 1788588040.204338 read 0x01a821 thru 0x01aa2a (0x20a bytes) 000327[0] 1788588040.215129 write 0x3328b0 thru 0x333f5b (0x16ac bytes) 000328[0] 1788588040.261072 read 0x7c39dc thru 0x7c3bad (0x1d2 bytes) 000329[0] 1788588040.282278 mapread 0x254fdd thru 0x2553ab (0x3cf bytes) 000330[0] 1788588040.310169 trunc from 0x7e2de9 to 0x26e7b6 (0x574632 bytes) 000333[1] 1788588040.459952 mapread 0x21daab thru 0x21e36d (0x8c3 bytes) 000334[1] 1788588040.553618 trunc from 0x26e7b7 to 0x8b2446 (0x643c90 bytes) 000335[0] 1788588040.625426 mapread 0x629351 thru 0x62a788 (0x1438 bytes) 000337[1] 1788588040.687986 mapwrite 0x7741e1 thru 0x77556c (0x138c bytes) 000338[1] 1788588040.770458 mapwrite 0x14d3d4 thru 0x14dbb4 (0x7e1 bytes) 000339[0] 1788588040.833793 mapread 0x1038ea thru 0x10484a (0xf61 bytes) 000340[0] 1788588040.884906 write 0x19b6b1 thru 0x19c1f0 (0xb40 bytes) 000342[1] 1788588040.940470 mapread 0x6119d0 thru 0x612bfa (0x122b bytes) 000343[0] 1788588040.993997 mapread 0x1355df thru 0x135c5b (0x67d bytes) 000345[0] 1788588041.050993 punch from 0x571b86 to 0x572bb4 (0x102f bytes) 000347[0] 1788588041.109752 write 0x05e231 thru 0x05eb92 (0x962 bytes) 000348[0] 1788588041.127159 mapwrite 0x847cf9 thru 0x849afe (0x1e06 bytes) 000349[0] 1788588041.246708 read 0x700716 thru 0x701433 (0xd1e bytes) 000350[0] 1788588041.295287 punch from 0x38fe6e to 0x3915cf (0x1762 bytes) 000351[1] 1788588041.302648 mapwrite 0x46bff0 thru 0x46c3dc (0x3ed bytes) 000352[0] 1788588041.426526 mapwrite 0x9fbedd thru 0x9fd453 (0x1577 bytes) 000355[1] 1788588041.575920 punch from 0x8fc75e to 0x8fdc29 (0x14cc bytes) 000358[0] 1788588041.662969 mapwrite 0x3ef688 thru 0x3f0141 (0xaba bytes) 000361[1] 1788588041.772298 mapread 0x6e810a thru 0x6e88f3 (0x7ea bytes) 000362[1] 1788588041.809081 mapwrite 0x83ef2a thru 0x83fecf (0xfa6 bytes) 000364[0] 1788588041.941332 read 0x6b9181 thru 0x6b926b (0xeb bytes) 000365[1] 1788588041.975790 mapread 0x45637a thru 0x45836b (0x1ff2 bytes) 000367[1] 1788588042.008729 read 0x16ad59 thru 0x16b0a8 (0x350 bytes) 000368[1] 1788588042.018273 trunc from 0x9fd454 to 0x2056d0 (0x7f7d83 bytes) 000369[0] 1788588042.075437 mapread 0x148f60 thru 0x149225 (0x2c6 bytes) 000370[1] 1788588042.111394 mapwrite 0x137d8f thru 0x137e79 (0xeb bytes) 000371[1] 1788588042.242187 mapread 0x17606f thru 0x177fa2 (0x1f34 bytes) 000373[1] 1788588042.258964 write 0x3eb05a thru 0x3eceb6 (0x1e5d bytes) 000375[1] 1788588042.283772 punch from 0x27a5ad to 0x27b073 (0xac7 bytes) 000376[1] 1788588042.309559 punch from 0x1fcd42 to 0x1fea79 (0x1d38 bytes) 000377[0] 1788588042.314692 mapread 0x377565 thru 0x37764f (0xeb bytes) 000378[1] 1788588042.367088 mapread 0x3acf85 thru 0x3ad490 (0x50c bytes) 000379[0] 1788588042.380389 trunc from 0x3eceb7 to 0x06d0e9 (0x37fdcd bytes) 000380[0] 1788588042.487024 write 0x0cb394 thru 0x0cb46c (0xd9 bytes) 000381[1] 1788588042.495224 punch from 0x023d04 to 0x024b2a (0xe27 bytes) 000383[0] 1788588042.558638 punch from 0x09db37 to 0x09eddc (0x12a6 bytes) 000384[1] 1788588042.578506 read 0x07390d thru 0x0757fd (0x1ef1 bytes) 000386[1] 1788588042.663356 mapread 0x0ae859 thru 0x0af4ba (0xc62 bytes) 000387[1] 1788588042.691100 punch from 0x0422be to 0x042f59 (0xc9c bytes) 000388[0] 1788588042.748745 punch from 0x07f68c to 0x0815b8 (0x1f2d bytes) 000390[1] 1788588042.775102 punch from 0x0ad06b to 0x0ad893 (0x829 bytes) 000391[0] 1788588042.807076 trunc from 0x0cb46d to 0x260b5c (0x1956f0 bytes) 000392[0] 1788588042.844858 read 0x145bda thru 0x146fa8 (0x13cf bytes) 000393[0] 1788588042.919967 read 0x036574 thru 0x037f77 (0x1a04 bytes) 000394[1] 1788588042.920550 punch from 0x078c3f to 0x07a710 (0x1ad2 bytes) 000395[0] 1788588042.954129 read 0x049c85 thru 0x04a9fb (0xd77 bytes) 000399[1] 1788588043.007319 mapwrite 0x64243e thru 0x643e63 (0x1a26 bytes) 000401[0] 1788588043.296761 punch from 0x3749e0 to 0x37698c (0x1fad bytes) 000402[0] 1788588043.338741 write 0x02e141 thru 0x02f3dc (0x129c bytes) 000404[1] 1788588043.420983 mapread 0x5e9738 thru 0x5eab03 (0x13cc bytes) 000405[0] 1788588043.443655 mapwrite 0x7e39b7 thru 0x7e3a23 (0x6d bytes) 000406[0] 1788588043.589556 mapread 0x60718a thru 0x608ccb (0x1b42 bytes) 000410[1] 1788588043.646029 mapread 0x52f359 thru 0x530e3d (0x1ae5 bytes) 000411[1] 1788588043.711617 trunc from 0x7e3a24 to 0x90866b (0x124c48 bytes) 000413[0] 1788588043.801160 mapwrite 0x3c9cfd thru 0x3c9fd8 (0x2dc bytes) 000415[0] 1788588043.902702 read 0x7caede thru 0x7cbc9b (0xdbe bytes) 000416[1] 1788588043.937188 read 0x21d235 thru 0x21f073 (0x1e3f bytes) 000418[1] 1788588043.956737 punch from 0x1df87b to 0x1dfd01 (0x487 bytes) 000419[0] 1788588043.971894 read 0x15f410 thru 0x15f85c (0x44d bytes) 000420[1] 1788588043.985680 mapread 0x00726b thru 0x007f07 (0xc9d bytes) 000422[1] 1788588044.002198 mapread 0x86c217 thru 0x86db8b (0x1975 bytes) 000423[1] 1788588044.024738 mapwrite 0x10fa3d thru 0x110b3c (0x1100 bytes) 000424[1] 1788588044.172937 punch from 0x0410a6 to 0x042783 (0x16de bytes) 000425[1] 1788588044.202546 mapwrite 0x79fca1 thru 0x7a0f47 (0x12a7 bytes) 000426[0] 1788588044.308963 mapread 0x38756d thru 0x389310 (0x1da4 bytes) 000427[0] 1788588044.388047 mapwrite 0x5fceb0 thru 0x5fcf44 (0x95 bytes) 000428[0] 1788588044.451801 mapread 0x3ff917 thru 0x3ffeb4 (0x59e bytes) 000429[0] 1788588044.478106 punch from 0x749e03 to 0x74a224 (0x422 bytes) 000430[1] 1788588044.486950 mapread 0x89ee6c thru 0x8a0b66 (0x1cfb bytes) 000431[0] 1788588044.512486 mapwrite 0x7cd47f thru 0x7cdbaf (0x731 bytes) 000432[1] 1788588044.576324 write 0x02cca8 thru 0x02eb1e (0x1e77 bytes) 000433[1] 1788588044.622798 mapread 0x6c03ee thru 0x6c0cf0 (0x903 bytes) 000434[1] 1788588044.658012 mapread 0x64cb4f thru 0x64e7e7 (0x1c99 bytes) 000436[1] 1788588044.717762 mapwrite 0x149c5d thru 0x14a8f2 (0xc96 bytes) 000437[1] 1788588044.814169 read 0x72401b thru 0x7251b2 (0x1198 bytes) 000439[0] 1788588044.872473 mapread 0x3d3f07 thru 0x3d5d47 (0x1e41 bytes) 000440[1] 1788588044.933478 punch from 0x67ae3c to 0x67c23e (0x1403 bytes) 000441[1] 1788588044.999793 read 0x5fe77d thru 0x5fefc6 (0x84a bytes) 000443[1] 1788588045.007730 write 0x031103 thru 0x0328bb (0x17b9 bytes) 000444[0] 1788588045.095652 mapread 0x15199e thru 0x152e2e (0x1491 bytes) 000445[1] 1788588045.194205 write 0x6db496 thru 0x6dd16b (0x1cd6 bytes) 000448[1] 1788588045.293870 mapread 0x23b61f thru 0x23c81d (0x11ff bytes) 000450[1] 1788588045.318448 trunc from 0x90866c to 0x40950e (0x4ff15d bytes) 000451[0] 1788588045.382514 mapwrite 0x8924fd thru 0x8936cd (0x11d1 bytes) 000456[0] 1788588045.640254 read 0x26d552 thru 0x26d96c (0x41b bytes) 000457[0] 1788588045.675419 mapread 0x1ff312 thru 0x1ff68b (0x37a bytes) 000459[1] 1788588045.707251 mapread 0x644566 thru 0x644718 (0x1b3 bytes) 000460[1] 1788588045.762385 mapwrite 0x2b66cd thru 0x2b6e21 (0x755 bytes) 000462[1] 1788588045.848035 read 0x675fe8 thru 0x676fa8 (0xfc1 bytes) 000463[1] 1788588045.870282 mapwrite 0x70830f thru 0x708751 (0x443 bytes) 000464[1] 1788588046.003706 write 0x491424 thru 0x492493 (0x1070 bytes) 000465[0] 1788588046.054271 punch from 0x586971 to 0x588257 (0x18e7 bytes) 000466[0] 1788588046.130266 mapread 0x09b49f thru 0x09c9fc (0x155e bytes) 000467[1] 1788588046.170757 read 0x1578d7 thru 0x158182 (0x8ac bytes) 000468[1] 1788588046.179062 mapread 0x1639b7 thru 0x1645ab (0xbf5 bytes) 000469[1] 1788588046.206877 read 0x4a28c6 thru 0x4a43d6 (0x1b11 bytes) 000474[0] 1788588046.240312 read 0x47375c thru 0x474b16 (0x13bb bytes) 000475[0] 1788588046.251034 read 0x247a6f thru 0x248313 (0x8a5 bytes) 000476[1] 1788588046.259207 trunc from 0x8936ce to 0x8bd191 (0x29ac4 bytes) 000477[1] 1788588046.379794 read 0x5c6285 thru 0x5c757f (0x12fb bytes) 000478[1] 1788588046.429130 write 0x63564d thru 0x6362ea (0xc9e bytes) 000482[0] 1788588046.563400 mapread 0x09aa0c thru 0x09c52a (0x1b1f bytes) 000484[1] 1788588046.672635 mapwrite 0x18a953 thru 0x18bc2f (0x12dd bytes) 000485[0] 1788588046.816697 punch from 0x494cd8 to 0x495f13 (0x123c bytes) 000486[0] 1788588046.847803 mapwrite 0x2cab49 thru 0x2cb21e (0x6d6 bytes) 000487[0] 1788588046.960304 write 0x42f0d9 thru 0x430ee0 (0x1e08 bytes) 000488[0] 1788588046.993407 trunc from 0x8bd192 to 0x81c0d3 (0xa10be bytes) 000489[1] 1788588047.025433 trunc from 0x81c0d4 to 0x5150d1 (0x307002 bytes) 000490[1] 1788588047.090337 write 0x89970e thru 0x89a3c4 (0xcb7 bytes) 000493[0] 1788588047.094914 read 0x474f96 thru 0x476d8b (0x1df6 bytes) 000496[1] 1788588047.117074 write 0x7fdd8c thru 0x7fe366 (0x5db bytes) 000497[1] 1788588047.134370 write 0x1cb1a2 thru 0x1cb739 (0x598 bytes) 000498[0] 1788588047.139715 mapread 0x324c49 thru 0x3255b5 (0x96d bytes) 000499[0] 1788588047.210254 write 0x3da53b thru 0x3db45c (0xf22 bytes) All operations completed A-OK! PASS 16b (31s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 02:00:52 (1788588052) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 02:00:57 (1788588057) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.434983 s, 19.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.447731 s, 18.7 MB/s ldlm.namespaces.MGC192.168.206.102@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b888624e800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b8893a71800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8b888624e800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8b8893a71800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b8893a71800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b8893a71800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.492 s, 8.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.23942 s, 24.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.8939 s, 36.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.10057 s, 17.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.62872 s, 12.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.02758 s, 34.6 MB/s PASS 16d (51s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 02:01:48 (1788588108) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.343155 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106895 s, 9.8 MB/s PASS 16e (7s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 02:01:55 (1788588115) TIMEOUT PASS 16f (27s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 02:02:22 (1788588142) TIMEOUT PASS 16g (29s) == sanityn test 16h: mmap read after truncate file ======= 02:02:51 (1788588171) striped dir -i0 -c2 -H all_char /mnt/lustre/d16h.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (8s) == sanityn test 16i: read after truncate file ============ 02:02:59 (1788588179) striped dir -i0 -c2 -H crush /mnt/lustre/d16i.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (7s) == sanityn test 16j: race dio with buffered i/o ========== 02:03:06 (1788588186) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.13598 s, 23.1 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.1815 s, 22.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.53677 s, 10.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.7788 s, 14.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.47334 s, 10.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.09338 s, 12.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.90172 s, 9.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.21785 s, 21.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.56306 s, 16.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.16564 s, 8.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.36726 s, 7.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.25595 s, 11.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.35669 s, 11.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.89343 s, 9.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.13467 s, 8.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.31007 s, 7.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.49171 s, 7.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.56628 s, 7.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.75419 s, 7.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (39s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 02:03:45 (1788588225) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! All operations completed A-OK! Sat Sep 5 02:04:32 EDT 2026 + fsx 2 done: rc=0 Sat Sep 5 02:04:32 EDT 2026 + fsx 3 done: rc=0 Sat Sep 5 02:04:32 EDT 2026 All operations completed A-OK! + fsx 4 done: rc=0 Sat Sep 5 02:04:32 EDT 2026 + fsx 5 done: rc=0 Sat Sep 5 02:04:32 EDT 2026 PASS 16k (57s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 02:04:42 (1788588282) fail_loc=0x8000030a PASS 17 (11s) == sanityn test 18: mmap sanity check =========================================================================================== 02:04:53 (1788588293) mmap test1: basic mmap operation (PASS, 0.16994s) mmap test2: MAP_PRIVATE not write back (PASS, 0.097766s) mmap test3: concurrent mmap ops on two nodes (PASS, 13.099s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.5563s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.15618s) mmap test6: check mmap write/read content on two nodes (PASS, 0.23772s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.2367s) mmap test11: multi-thread shared mmap (PASS, 1.0674s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1s) PASS 18 (30s) == sanityn test 19: test concurrent uncached read races ========================================================================= 02:05:23 (1788588323) oleg602-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg602-client: oleg602-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 02:05:27 (1788588327) striped dir -i0 -c2 -H all_char /mnt/lustre/d20.sanityn PASS 20 (8s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 02:05:35 (1788588335) striped dir -i1 -c2 -H crush2 /mnt/lustre/d21.sanityn rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (8s) == sanityn test 23: others should see updated atime while another read============================================================== 02:05:43 (1788588343) now is 1788588346 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.8428 new atime is 1788588407 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 02:06:53 (1788588413) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 7168 3762176 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 516.2K 405 515.8K 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 518.7K 335 518.4K 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 114.5K 467 114.1K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 115.3K 466 114.9K 1% /mnt/lustre[OST:1] filesystem_summary: 229.7K 740 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 516.2K 405 515.8K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 518.7K 335 518.4K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 114.5K 467 114.1K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 115.3K 466 114.9K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.7K 740 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.4M 2.1G 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 7.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 39.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 528612 405 528207 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 531165 335 530830 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 117299 467 116832 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 118098 466 117632 1% /mnt/lustre2[OST:1] filesystem_summary: 235204 740 234464 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 516.2K 405 515.8K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 518.7K 335 518.4K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 114.5K 467 114.1K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 115.3K 466 114.9K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.7K 740 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 528612 405 528207 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 531165 335 530830 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 117299 467 116832 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118098 466 117632 1% /mnt/lustre[OST:1] filesystem_summary: 235204 740 234464 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 528612 405 528207 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 531165 335 530830 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 117299 467 116832 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 118098 466 117632 1% /mnt/lustre2[OST:1] filesystem_summary: 235204 740 234464 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 7168 3762176 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 PASS 24a (6s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 02:06:59 (1788588419) PASS 24b (7s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 02:07:06 (1788588426) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (8s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 02:07:14 (1788588434) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] PASS 25b (7s) == sanityn test 26a: allow mtime to get older ============ 02:07:21 (1788588441) utime: good mknod times 1788588442 <= 1788588443 <= 1788588443 for /mnt/lustre/f26a utime: good mknod times 1788588442 <= 1788588443 <= 1788588443 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (8s) == sanityn test 26b: sync mtime between ost and mds ====== 02:07:29 (1788588449) PASS 26b (9s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 02:07:38 (1788588458) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8428 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279602 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-05 02:07:40.000000000 -0400 Birth: 2026-09-05 02:07:40.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279602 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-05 02:07:40.000000000 -0400 Birth: 2026-09-05 02:07:40.000000000 -0400 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 02:07:48 (1788588468) striped dir -i0 -c2 -H crush2 /mnt/lustre/d30.sanityn 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 02:07:57 (1788588477) striped dir -i1 -c2 -H all_char /mnt/lustre/d31a.sanityn fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 02:08:05 (1788588485) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg602-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete striped dir -i1 -c2 -H all_char /mnt/lustre/d31b.sanityn fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.206.102@tcp:/lustre 7542784 40960 7495680 1% /mnt/lustre2 PASS 31b (22s) == sanityn test 31r: open-rename(replace) race =========== 02:08:27 (1788588507) -rw-r--r-- 1 root root 0 Sep 5 02:08 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 02:08:37 (1788588517) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 45188 Terminated tail -f $tfile PASS 31s (7s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 02:08:44 (1788588524) PASS 31t (8s) == sanityn test 32b: lockless i/o ======================== 02:08:52 (1788588532) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (3s) == sanityn test 32c: contention detection ================ 02:08:55 (1788588535) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.78218 s, 29.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.59002 s, 33.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.06702 s, 25.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.26569 s, 23.1 MB/s contention_seconds 4, contention_events 18 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.8845 s, 27.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.1127 s, 24.8 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.21989 s, 43.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.33725 s, 39.2 MB/s PASS 32c (30s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 02:09:27 (1788588567) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 02:09:30 (1788588570) Stopping /mnt/lustre-mds1 (opts:) on oleg602-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg602-server: oleg602-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg602-client: oleg602-server: ssh exited with exit code 1 Started lustre-MDT0000 create file on 2nd stripe under striped directory /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre/d33c.sanityn/striped touch /mnt/lustre2/d33c.sanityn/striped/subfile SoLC count 2 Below operations shouldn't trigger Solc: create file on 2nd stripe after setattr chmod 777 /mnt/lustre/d33c.sanityn/striped; touch /mnt/lustre2/d33c.sanityn/striped/subfile create remote dir and local dir /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote create local dir after remote dir creation transaction commit /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote.3; do_facet mds2 /home/green/git/lustre-release/lustre/utils/lctl set_param -n osd*.*MDT0001.force_sync 1; mkdir /mnt/lustre/d33c.sanityn/local.3 Solc count 0 PASS 33c (59s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 02:10:29 (1788588629) remote directory create /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre2/d33d.sanityn/subdir CoS count 2 cross-MDT rename mkdir /mnt/lustre/d33d.sanityn; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn/d1; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33d.sanityn/d2; touch /mnt/lustre/d33d.sanityn/d1/src mv /mnt/lustre2/d33d.sanityn/d1/src /mnt/lustre2/d33d.sanityn/d2/tgt CoS count 1 migrate /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs migrate -m 1 /mnt/lustre2/d33d.sanityn CoS count 2 PASS 33d (66s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 02:11:36 (1788588696) plain dir creation mkdir /mnt/lustre2/d33e.sanityn/plain open file and write echo abc > /mnt/lustre2/d33e.sanityn/f33e.sanityn append write echo dfg >> /mnt/lustre2/d33e.sanityn/f33e.sanityn setattr touch /mnt/lustre2/d33e.sanityn/f33e.sanityn file unlink rm /mnt/lustre2/d33e.sanityn/f33e.sanityn plain dir unlink rmdir /mnt/lustre2/d33e.sanityn/plain striped directory creation /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre2/d33e.sanityn/striped set default LMV to create striped subdir /home/green/git/lustre-release/lustre/utils/lfs setdirstripe -D -c 2 /mnt/lustre/d33e.sanityn striped subdir creation createmany -d /mnt/lustre/d33e.sanityn/subdir 100 total: 100 mkdir in 1.49 seconds: 66.91 ops/second sub file creation and write createmany -o /mnt/lustre/d33e.sanityn/subfile 100; echo abc > /mnt/lustre/d33e.sanityn/subfile1 total: 100 open/close in 1.38 seconds: 72.35 ops/second sub file append write echo dfg >> /mnt/lustre2/d33e.sanityn/subfile2 subdir setatttr touch /mnt/lustre2/d33e.sanityn/subdir1 subdir unlink unlinkmany -d /mnt/lustre/d33e.sanityn/subdir 100 - unlinked 0 (time 1788588703 ; total 1 ; last 1) total: 100 unlinks in 6 seconds: 16.666666 unlinks/second sub file unlink unlinkmany /mnt/lustre2/d33e.sanityn/subfile 100 - unlinked 0 (time 1788588708 ; total 0 ; last 0) total: 100 unlinks in 3 seconds: 33.333332 unlinks/second sub file creation follows striped dir chmod chmod 777 /mnt/lustre/d33e.sanityn/striped; touch /mnt/lustre/d33e.sanityn/striped/subfile striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn/striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn CoS count 0 PASS 33e (23s) == sanityn test 34: no lock timeout under IO ============= 02:11:58 (1788588718) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg602-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b888624e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b888624e800.ost_server_uuid in IDLE state after 0 sec oleg602-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b888624e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b888624e800.ost_server_uuid in IDLE state after 0 sec oleg602-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b888624e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b888624e800.ost_server_uuid in IDLE state after 0 sec oleg602-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b888624e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b888624e800.ost_server_uuid in IDLE state after 0 sec oleg602-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b888624e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b888624e800.ost_server_uuid in IDLE state after 0 sec oleg602-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b888624e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b888624e800.ost_server_uuid in IDLE state after 0 sec PASS 34 (138s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 02:14:16 (1788588856) striped dir -i1 -c2 -H all_char /mnt/lustre/d35.sanityn fail_loc=0x80000317 Race attempt 0 Wait for 57217 57250 for 60 sec... - open/close 702 (time 1788588873.62 total 10.01 last 70.14) - open/close 1370 (time 1788588883.62 total 20.01 last 66.79) - open/close 2066 (time 1788588893.63 total 30.02 last 69.52) - open/close 2759 (time 1788588903.63 total 40.03 last 69.25) - open/close 3449 (time 1788588913.64 total 50.03 last 68.95) total: 4000 open/close in 58.11 seconds: 68.84 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.206.102@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.206.102@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (73s) == sanityn test 36: handle ESTALE/open-unlink correctly == 02:15:29 (1788588929) striped dir -i0 -c2 -H crush2 /mnt/lustre/d36.sanityn Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg602-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.59804 s, 32.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8428 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.927777 s, 56.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8428 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.30115 s, 40.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8428 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.29176 s, 40.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8428 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.35037 s, 38.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8428 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.3949 s, 37.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8428 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7493632) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.53065 s, 34.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8428 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7490560) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.29026 s, 40.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8428 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7491584) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.16148 s, 45.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8428 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7493632) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.062 s, 25.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8428 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7491584) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.25174 s, 41.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8428 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(10) *** before(7497728) after_dd(7444480) after(7495680) PASS 36 (323s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 02:20:52 (1788589252) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d37.sanityn multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.8428 - create 1097 (time 1788589267.74 total 10.01 last 109.63) - create 2443 (time 1788589277.74 total 20.01 last 134.56) - create 3774 (time 1788589287.74 total 30.01 last 133.08) - create 5133 (time 1788589297.74 total 40.01 last 135.89) - create 6468 (time 1788589307.75 total 50.02 last 133.46) - create 7935 (time 1788589317.75 total 60.02 last 146.69) - create 9319 (time 1788589327.75 total 70.02 last 138.39) total: 10000 create in 74.84 seconds: 133.63 ops/second PASS 37 (86s) == sanityn test 39a: file mtime does not change after rename ========================================================== 02:22:18 (1788589338) repeat after cancel_lru_locks PASS 39a (6s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 02:22:24 (1788589344) repeat after cancel_lru_locks PASS 39b (7s) == sanityn test 39c: check truncate mtime update ================================================================================ 02:22:31 (1788589351) repeat after cancel_lru_locks PASS 39c (7s) == sanityn test 39d: sync write should update mtime ====== 02:22:38 (1788589358) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 02:22:47 (1788589367) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (17s) == sanityn test 40c: pdirops: link and others ======================================================================== 02:23:04 (1788589384) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (17s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 02:23:21 (1788589401) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (16s) == sanityn test 40e: pdirops: rename and others ======================================================================== 02:23:37 (1788589417) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (16s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 02:23:53 (1788589433) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (13s) == sanityn test 41b: pdirops: create vs create ======================================================================== 02:24:06 (1788589446) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 41b (19s) == sanityn test 41c: pdirops: create vs link ======================================================================== 02:24:25 (1788589465) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 02:24:39 (1788589479) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41d (12s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 02:24:51 (1788589491) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41e (13s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 02:25:04 (1788589504) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41f (13s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 02:25:17 (1788589517) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41g (13s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 02:25:30 (1788589530) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (13s) == sanityn test 41i: reint_open: create vs create ======== 02:25:43 (1788589543) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1074s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 02:43:38 (1788590618) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (16s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 02:43:53 (1788590633) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 42b (14s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 02:44:07 (1788590647) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (14s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 02:44:21 (1788590661) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42d (16s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 02:44:37 (1788590677) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (15s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 02:44:52 (1788590692) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42f (17s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 02:45:09 (1788590709) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 42g (13s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 02:45:22 (1788590722) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42h (16s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 02:45:38 (1788590738) PASS 43a (211s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 02:49:09 (1788590949) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43b (13s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 02:49:22 (1788590962) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43c (14s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 02:49:37 (1788590977) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (13s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 02:49:50 (1788590990) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43e (15s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 02:50:05 (1788591005) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (15s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 02:50:20 (1788591020) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (16s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 02:50:36 (1788591036) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43h (15s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 02:50:52 (1788591052) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43i (16s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 02:51:07 (1788591067) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (131s) == sanityn test 43k: unlink vs create ==================== 02:53:18 (1788591198) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1333s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 03:15:31 (1788592531) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Sat Sep 5 03:15:40 EDT 2026 PASS 44a (15s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 03:15:46 (1788592546) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 44b (15s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 03:16:01 (1788592561) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (15s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 03:16:16 (1788592576) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44d (14s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 03:16:30 (1788592590) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44e (14s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 03:16:44 (1788592604) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44f (14s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 03:16:58 (1788592618) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44g (14s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 03:17:12 (1788592632) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44h (15s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 03:17:27 (1788592647) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 lfs mkdir: dirstripe error on '/mnt/lustre2/f44i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f44i.sanityn': File exists Conflict PASS 44i (13s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 03:17:40 (1788592660) PASS 45a (245s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 03:21:45 (1788592905) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45b (15s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 03:22:00 (1788592920) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45c (15s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 03:22:15 (1788592935) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (15s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 03:22:30 (1788592950) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45e (15s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 03:22:45 (1788592965) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (13s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 03:22:58 (1788592978) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (14s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 03:23:12 (1788592992) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45h (13s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 03:23:26 (1788593006) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45i (14s) == sanityn test 45j: read vs rename ====================== 03:23:40 (1788593020) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1301s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 03:45:21 (1788594321) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (13s) == sanityn test 46b: pdirops: link vs create ======================================================================== 03:45:34 (1788594334) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 46b (15s) == sanityn test 46c: pdirops: link vs link ======================================================================== 03:45:49 (1788594349) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (13s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 03:46:02 (1788594362) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46d (14s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 03:46:16 (1788594376) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46e (12s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 03:46:29 (1788594389) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46f (14s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 03:46:43 (1788594403) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46g (12s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 03:46:56 (1788594416) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46h (14s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 03:47:09 (1788594429) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 lfs mkdir: dirstripe error on '/mnt/lustre2/f46i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f46i.sanityn': File exists Conflict PASS 46i (14s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 03:47:23 (1788594443) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f47a.sanityn': File exists Conflict PASS 47a (16s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 03:47:39 (1788594459) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 47b (15s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 03:47:54 (1788594474) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f47c.sanityn' to '/mnt/lustre2/f47c.sanityn-2': File exists Conflict PASS 47c (14s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 03:48:08 (1788594488) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 47d (12s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 03:48:20 (1788594500) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f47e.sanityn' with non-directory Conflict PASS 47e (12s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 03:48:32 (1788594512) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 47f (12s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 03:48:44 (1788594524) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 47g (18s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 03:49:02 (1788594542) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.161662 s, 63.3 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (12s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 03:49:14 (1788594554) 0+1 records in 0+1 records out 158 bytes copied, 0.0332015 s, 4.8 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (237033) - No such process PASS 51a (9s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 03:49:23 (1788594563) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0180174 s, 56.8 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713379 s, 14.7 MB/s 3145728 PASS 51b (20s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 03:49:44 (1788594584) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0438753 s, 23.3 kB/s PASS 51c (13s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 03:49:57 (1788594597) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391619 s, 26.8 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (8s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 03:50:05 (1788594605) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 1290 0x50a 0x2c0000401 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 1290 0x50a 0x2c0000401 read lease(1) has applied. PASS 51e (8s) == sanityn test 54: rename locking ======================= 03:50:13 (1788594613) llite.lustre-ffff8b888624e800.xattr_cache=1 llite.lustre-ffff8b8893a71800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762680 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 03:50:14.000000000 -0400 Modify: 2026-09-05 03:50:15.000000000 -0400 Change: 2026-09-05 03:50:15.000000000 -0400 Birth: 2026-09-05 03:50:14.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281266 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 03:50:21.000000000 -0400 Modify: 2026-09-05 03:50:22.000000000 -0400 Change: 2026-09-05 03:50:22.000000000 -0400 Birth: 2026-09-05 03:50:21.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762683 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 03:50:28.000000000 -0400 Modify: 2026-09-05 03:50:28.000000000 -0400 Change: 2026-09-05 03:50:28.000000000 -0400 Birth: 2026-09-05 03:50:27.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281267 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 03:50:27.000000000 -0400 Modify: 2026-09-05 03:50:29.000000000 -0400 Change: 2026-09-05 03:50:29.000000000 -0400 Birth: 2026-09-05 03:50:27.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281269 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 03:50:35.000000000 -0400 Modify: 2026-09-05 03:50:36.000000000 -0400 Change: 2026-09-05 03:50:36.000000000 -0400 Birth: 2026-09-05 03:50:35.000000000 -0400 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762684 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 03:50:35.000000000 -0400 Modify: 2026-09-05 03:50:36.000000000 -0400 Change: 2026-09-05 03:50:36.000000000 -0400 Birth: 2026-09-05 03:50:35.000000000 -0400 PASS 54 (34s) == sanityn test 55a: rename vs unlink target dir ========= 03:50:47 (1788594647) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (13s) == sanityn test 55b: rename vs unlink source dir ========= 03:51:00 (1788594660) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (13s) == sanityn test 55c: rename vs unlink orphan target dir == 03:51:13 (1788594673) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (19s) == sanityn test 55d: rename file vs link ================= 03:51:32 (1788594692) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (15s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 03:51:47 (1788594707) fail_loc=0x155 We survived after AB/BA race test fail_loc=0x0 PASS 55e (21s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 03:52:08 (1788594728) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (20s) == sanityn test 55g: rename: race with trylock =========== 03:52:28 (1788594748) mdt.lustre-MDT0000.enable_rename_trylock=1 mdt.lustre-MDT0001.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (22s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 03:52:51 (1788594771) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788594772 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.616556 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (107s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 03:54:38 (1788594878) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788594883 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.0803 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (128s) == sanityn test 60: Verify data_version behaviour ======== 03:56:46 (1788595006) striped dir -i0 -c2 -H all_char /mnt/lustre/d60.sanityn PASS 60 (8s) == sanityn test 70a: cd directory && rm directory ======== 03:56:54 (1788595014) PASS 70a (8s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 03:57:02 (1788595022) PASS 70b (9s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 03:57:11 (1788595031) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00660933 s, 9.9 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 03:57:15 (1788595035) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0225116 s, 2.9 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 03:57:18 (1788595038) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 03:57:21 (1788595041) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0233968 s, 2.8 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (3s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 03:57:24 (1788595044) llite.lustre-ffff8b888624e800.xattr_cache=1 llite.lustre-ffff8b8893a71800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (7s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 03:57:31 (1788595051) llite.lustre-ffff8b888624e800.xattr_cache=1 llite.lustre-ffff8b8893a71800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (7s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 03:57:38 (1788595058) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0155153 s, 66.0 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0123567 s, 82.9 kB/s 250335: get lock1 250334: get lock2 250335: done 250335: get lock2 250334: done 250334: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 250334: cannot get lock1: Resource deadlock avoided 250335: done 250335: exit rc=0 250334: done 250334: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 03:57:49 (1788595069) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0576709 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286302 s, 36.6 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 250924 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (12s) == sanityn test 76: Verify MDT open_files listing ======== 03:58:01 (1788595081) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d76.sanityn opening files: ................ closing files: ................ PASS 76 (182s) == sanityn test 77a: check FIFO NRS policy =============== 04:01:03 (1788595263) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.570943 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117877 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120066 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18401 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117107 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10802 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198931 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183494 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12789 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158991 s, 6.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347114 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199286 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31533 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351584 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377992 s, 2.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437287 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.503874 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685242 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140522 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157076 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158438 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909178 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16761 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170632 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172299 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17575 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10998 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163405 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144368 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18868 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192802 s, 5.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200853 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244039 s, 4.3 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg602-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg602-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (11s) == sanityn test 77b: check CRR-N NRS policy ============== 04:01:14 (1788595274) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.455508 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070069 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120884 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563464 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167674 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171152 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168314 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179792 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200279 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147579 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28168 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313263 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295811 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335207 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337173 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325424 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00602833 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158793 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00506795 s, 207 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384838 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167199 s, 62.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00477639 s, 220 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00362147 s, 290 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188176 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202956 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167767 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129987 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864123 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00433429 s, 242 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060212 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232921 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154987 s, 67.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126727 s, 82.7 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg602-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg602-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.391322 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897553 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134173 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12121 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153072 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153284 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160403 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543608 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168679 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135076 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12185 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170544 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592827 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113827 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195092 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19638 s, 5.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223092 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771196 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326458 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406039 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508172 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441527 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885085 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492543 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692868 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688987 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561982 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671336 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729979 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862418 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614459 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662704 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758777 s, 13.8 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (12s) == sanityn test 77c: check ORR NRS policy ================ 04:01:26 (1788595286) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.324545 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521237 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365719 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695293 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217569 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23567 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244775 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21111 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209898 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242476 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235228 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260208 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262298 s, 4.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270238 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304681 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282976 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298768 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906791 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00513428 s, 204 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552987 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679572 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047605 s, 22.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676676 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122679 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1206 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125715 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100338 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141756 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910748 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134441 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276932 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230267 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987746 s, 10.6 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg602-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg602-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.381149 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627954 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763245 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203917 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186472 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22873 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246178 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249881 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285112 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252478 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260729 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30853 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306056 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31875 s, 3.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293332 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321288 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314603 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055695 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331154 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215266 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322549 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058805 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515046 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608275 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537958 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105869 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057386 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659824 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658269 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800737 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838641 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696024 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782269 s, 13.4 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.548857 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640624 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857947 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137981 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225908 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24872 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205932 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260782 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297434 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295281 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319965 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317575 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311608 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352126 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370604 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409064 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33205 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118856 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138547 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583092 s, 18.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101338 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134579 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795894 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100005 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168423 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103651 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16077 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100338 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131039 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891019 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878649 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092301 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123844 s, 8.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (16s) == sanityn test 77d: check TRR nrs policy ================ 04:01:43 (1788595303) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.60791 s, 27.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561441 s, 18.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931779 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245662 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249527 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300413 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2695 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316238 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300332 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269195 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319012 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337831 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389434 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407935 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417687 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374569 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390521 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515914 s, 20.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109229 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548896 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732808 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077306 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732473 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668831 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053138 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107146 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105803 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101952 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966299 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981036 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118589 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763496 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862191 s, 12.2 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.749664 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582619 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691756 s, 15.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705342 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108873 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128544 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208366 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232863 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259506 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303184 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302274 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23456 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250446 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30409 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230412 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211128 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27165 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763955 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541864 s, 19.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587918 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066011 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745035 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610799 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706513 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05511 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658767 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693666 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818186 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852054 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110352 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843337 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957927 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120127 s, 8.7 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg602-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg602-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.506246 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052188 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105705 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119542 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257096 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281358 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324322 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248676 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283157 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269388 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330295 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265962 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276052 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289772 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268993 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305837 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419308 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304354 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139049 s, 75.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570215 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261309 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660247 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405662 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07119 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664599 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424905 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393377 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771673 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396721 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567495 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522872 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541888 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577691 s, 18.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=5 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg602-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg602-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (18s) == sanityn test 77e: check TBF NID nrs policy ============ 04:02:01 (1788595321) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.206.2@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.440892 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101443 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101205 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248627 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296861 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278623 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109894 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109215 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37177 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362246 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119608 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386079 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398574 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158914 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355814 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179533 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.466692 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.445091 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39312 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.413818 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.487516 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197737 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00570167 s, 184 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112295 s, 93.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.523785 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108706 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10796 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433017 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307674 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372076 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388917 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187775 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117002 s, 89.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.410294 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946505 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971695 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199112 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14979 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982468 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19212 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197885 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314663 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368026 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335706 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285422 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296757 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354215 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.432297 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420651 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.478193 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115466 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132505 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173261 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105068 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854293 s, 12.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987442 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13646 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943162 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146772 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859279 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165597 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111363 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115055 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153549 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154776 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203212 s, 5.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.566794 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794119 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12755 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135442 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127004 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231491 s, 4.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276633 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231621 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242673 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270079 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302479 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310373 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391845 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359237 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421087 s, 2.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289562 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373826 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157899 s, 66.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00370764 s, 283 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172002 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12285 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112309 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00521236 s, 201 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125596 s, 83.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155593 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942971 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682419 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918766 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745625 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633525 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495871 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053441 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922541 s, 11.4 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg602-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg602-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.523917 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486558 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053798 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640781 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1031 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139465 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11113 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141549 s, 7.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104828 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100647 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610901 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255742 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238233 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145522 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290054 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263097 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22102 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373281 s, 28.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380977 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307593 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0076444 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.002385 s, 440 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00470579 s, 223 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00303284 s, 346 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00714057 s, 147 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190174 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989078 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635487 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677505 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053903 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215786 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903813 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767851 s, 13.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg602-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg602-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (24s) == sanityn test 77f: check TBF JobID nrs policy ========== 04:02:25 (1788595345) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg602-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.399335 s, 42.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044047 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068239 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746614 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934582 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971433 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171866 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132523 s, 7.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166592 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229239 s, 4.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205457 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22545 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205602 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244785 s, 4.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195666 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176255 s, 5.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284208 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00750515 s, 140 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00936402 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00627829 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238151 s, 44.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00388597 s, 270 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498759 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00459556 s, 228 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00635662 s, 165 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058477 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046954 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00515311 s, 203 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00438922 s, 239 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00514608 s, 204 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279507 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00404388 s, 259 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00473438 s, 221 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg602-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.450502 s, 37.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081163 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078889 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114022 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12765 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113422 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253808 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151933 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275609 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2016 s, 5.2 MB/s 1+0 records in 1+0 records out running as uid/gid/euid/egid 500/500/500/500, groups:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275338 s, 3.8 MB/s 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22794 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246647 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210398 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221633 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302181 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313507 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00838611 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505813 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807504 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00269843 s, 389 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112424 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302198 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126137 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372198 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00717411 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710776 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353595 s, 29.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132967 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688037 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13809 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303386 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171851 s, 6.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg602-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.507909 s, 33.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110224 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587577 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745949 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137158 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152142 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982429 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158016 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101896 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251124 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152229 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107818 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21024 s, 5.0 MB/s 1+0 records in 1+0 records out running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408102 s, 2.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274319 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247434 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325521 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023599 s, 44.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710174 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351456 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00532242 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013548 s, 77.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00989164 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00900431 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566397 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982681 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504413 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12754 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874078 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890161 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275563 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00972562 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250885 s, 41.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg602-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.426376 s, 39.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845717 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127163 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217092 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265763 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215787 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337756 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33804 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307326 s, 3.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268269 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227496 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330024 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253084 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256083 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301982 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302946 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352281 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370654 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914517 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425984 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788691 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540904 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462644 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558295 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460063 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119342 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528841 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130222 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732956 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790404 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818895 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854579 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870483 s, 12.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (22s) == sanityn test 77g: Change TBF type directly ============ 04:02:47 (1788595367) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (11s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 04:02:58 (1788595378) oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg602-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg602-client: oleg602-server: ssh exited with exit code 2 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg602-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg602-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg602-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 PASS 77h (12s) == sanityn test 77i: Change rank of TBF rule ============= 04:03:10 (1788595390) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (22s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 04:03:32 (1788595412) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.697212 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433153 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745092 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867037 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236786 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306648 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288078 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336718 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365663 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388315 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344899 s, 3.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344934 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409844 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394589 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408578 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369866 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431228 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29266 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232086 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229153 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242796 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287851 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309144 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241945 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274635 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259973 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256703 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249919 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300986 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327762 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276861 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291141 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323583 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=1 removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg602-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg602-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.01078 s, 11.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6187 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (67s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 04:04:39 (1788595479) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg602-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg602-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg602-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg602-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg602-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30164 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697869 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843264 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354127 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606052 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161495 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947666 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230183 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189397 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249122 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200012 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25935 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248624 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236814 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261927 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246582 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309914 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474897 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620761 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642977 s, 16.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672883 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665532 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726855 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600618 s, 17.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522012 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084296 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750372 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078547 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557209 s, 18.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916849 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804269 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670766 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745709 s, 14.1 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg602-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg602-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9047 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9773 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg602-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg602-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg602-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg602-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg602-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.31579 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839981 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101584 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956688 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12677 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137082 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107917 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152832 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180877 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185937 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302001 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453948 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144072 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192891 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257718 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252804 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276428 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164691 s, 63.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753541 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695467 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863068 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816626 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879449 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468757 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096723 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718485 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822804 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583965 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062984 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858618 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708917 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869043 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108971 s, 9.6 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg602-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg602-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9341 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9591 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (157s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 04:07:16 (1788595636) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg602-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg602-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg602-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg602-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg602-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.3592 s, 12.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19767 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221982 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12626 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150007 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166677 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264347 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270452 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297956 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308567 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265286 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291955 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377829 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338837 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346767 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398798 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34507 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739761 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769907 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420927 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858115 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607518 s, 17.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084362 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688737 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778535 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649583 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639553 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101472 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113895 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904605 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968952 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108678 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134183 s, 7.8 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1226 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7706 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg602-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg602-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg602-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg602-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg602-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27679 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111152 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138701 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107725 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196438 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199409 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289633 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298547 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354017 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348261 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359322 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379691 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351242 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387044 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3636 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390529 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381263 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454514 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424581 s, 24.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683095 s, 15.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680227 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269207 s, 39.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451132 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206092 s, 50.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077224 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445128 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728306 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054881 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104129 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710652 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757076 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118366 s, 88.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947881 s, 11.1 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg602-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg602-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9033 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9477 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (158s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 04:09:54 (1788595794) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg602-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.843713 s, 19.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710113 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954042 s, 11.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216614 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178599 s, 5.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28049 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269438 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321115 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388576 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.435591 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384252 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394262 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372109 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35939 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409239 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39384 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979951 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134913 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128617 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739055 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799454 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639183 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173812 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731283 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13677 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714113 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738135 s, 14.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912768 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694013 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110729 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.548044 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213245 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132224 s, 7.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.01675 s, 11.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88675 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.206.2@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.206.2@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.645795 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683817 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114089 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12477 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242641 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210991 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290577 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283124 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315555 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248948 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29746 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263047 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349859 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280793 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376903 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327377 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300249 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116358 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117567 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108233 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158195 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131407 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142848 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130911 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218785 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149817 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149165 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178604 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163803 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17763 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16639 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169833 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192313 s, 5.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=2 removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.72768 s, 12.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92976 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.206.2@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg602-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.593365 s, 28.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536163 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626953 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677369 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817756 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379356 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141758 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200591 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210357 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177563 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144439 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15433 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03805 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963841 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181206 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307198 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244968 s, 4.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=3 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607255 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111939 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119759 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595543 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626995 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699223 s, 15.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690273 s, 15.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804357 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961712 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103756 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162877 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120709 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131181 s, 8.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131662 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194308 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152426 s, 6.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6267 s, 9.9 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.52876 s, 19.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg602-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.704227 s, 23.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618183 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497843 s, 21.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12186 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143224 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226664 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205367 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244305 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250529 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281648 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298291 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300951 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31542 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339657 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319587 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.461027 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.452403 s, 2.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033657 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666397 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603624 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417893 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439191 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735154 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515505 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759212 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877127 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097811 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841085 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679326 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725022 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755454 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805011 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126707 s, 8.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg602-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg602-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.5261 s, 9.1 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93864 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.4904 s, 12.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98302 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg602-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.05814 s, 15.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726426 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539584 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116707 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1576 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174119 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251046 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281106 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276207 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289155 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290946 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280706 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294739 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26929 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30223 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316522 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340499 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456038 s, 23.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634025 s, 16.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501269 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483224 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052629 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882521 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118299 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538725 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570509 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550635 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100991 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104255 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119483 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126459 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955954 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113939 s, 9.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9094 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6602 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg602-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.921287 s, 18.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09825 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136977 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148897 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133058 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103029 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198491 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188697 s, 5.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324496 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393061 s, 2.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259825 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431772 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288021 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266201 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.504452 s, 2.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.546164 s, 1.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.700394 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160522 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196996 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175274 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182604 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157993 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136239 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114006 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136172 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12911 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16004 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270278 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187615 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187279 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270732 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127714 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152671 s, 6.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg602-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg602-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.7833 s, 8.9 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88429 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg602-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.660779 s, 25.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052523 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531862 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554281 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871607 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987276 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089271 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151668 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118898 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115504 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219774 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267868 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278609 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380224 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354851 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.463159 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468678 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206427 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265773 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278046 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434274 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764067 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767106 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431065 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403833 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410098 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104399 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643216 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986701 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569329 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753914 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626568 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685702 s, 15.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6448 s, 9.9 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97584 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6267 s, 9.9 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88711 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (449s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 04:17:24 (1788596244) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg602-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.697798 s, 24.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616226 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684396 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155606 s, 6.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184517 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22325 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235125 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266822 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279181 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300641 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252525 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305817 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253315 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302079 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322478 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286852 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348617 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459964 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00484921 s, 216 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813101 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00744035 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00598079 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150258 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995022 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164235 s, 63.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872108 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119167 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470934 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123243 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899294 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129803 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804674 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125625 s, 8.3 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.0771 s, 13.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95628 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (50s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 04:18:14 (1788596294) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.206.2@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.206.2@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.580698 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646958 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106613 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603038 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910612 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131988 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108681 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162808 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135712 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128526 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180577 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225014 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240046 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300174 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295206 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311074 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344803 s, 3.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108603 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114438 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123525 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133486 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12424 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686449 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134096 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133635 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152934 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732714 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134478 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14397 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10985 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154669 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905936 s, 11.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=12 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204805 s, 5.1 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.11696 s, 11.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95949 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (49s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 04:19:03 (1788596343) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.206.2@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg602-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.598494 s, 28.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424768 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563508 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805744 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744339 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164246 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203733 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178606 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18169 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21553 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231243 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203125 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229583 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253787 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262838 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256965 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272757 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b888624e800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=3 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530896 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559793 s, 18.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350303 s, 29.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057439 s, 18.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432289 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409328 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774202 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434631 s, 24.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663546 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532451 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530276 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499405 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580937 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527098 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571758 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656367 s, 16.0 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.29173 s, 14.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.02156 s, 20.9 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (45s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 04:19:49 (1788596389) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg602-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.758134 s, 22.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279336 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102553 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328959 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124039 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816026 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168663 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724649 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134055 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106297 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782419 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161236 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183341 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122477 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107112 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141845 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10735 s, 9.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593436 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117642 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495674 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669074 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715485 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659947 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459496 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139711 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849969 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913241 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878543 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109917 s, 9.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111878 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126015 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128642 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140719 s, 7.5 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.0536 s, 9.5 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9226 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.68483 s, 12.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9272 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (96s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 04:21:25 (1788596485) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg602-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.03044 s, 16.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806441 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739599 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121248 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200418 s, 5.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168091 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164478 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181267 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260178 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262984 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231849 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242097 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213742 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287494 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222537 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250398 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296868 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119052 s, 88.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419828 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049838 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384788 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762129 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887145 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612482 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743855 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772164 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991541 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931081 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135247 s, 77.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622561 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102238 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810863 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873202 s, 12.0 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8636 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9375 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (72s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 04:22:38 (1788596558) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg602-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.587182 s, 28.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678886 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524944 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109598 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154136 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207188 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172591 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194671 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252074 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261702 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237945 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256169 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241416 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233386 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243495 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296441 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327595 s, 3.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b888624e800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307395 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433546 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916779 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371206 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869397 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683374 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467614 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532781 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848648 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545444 s, 19.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088164 s, 11.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570102 s, 18.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907165 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603035 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715812 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614374 s, 17.1 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.43682 s, 12.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91473 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg602-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.731584 s, 22.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546217 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384995 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116977 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140048 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140111 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176794 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202094 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242778 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250161 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258079 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250656 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279174 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274918 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280737 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31188 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328038 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491403 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226988 s, 46.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285391 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351646 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00460016 s, 228 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870482 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784454 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720595 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259153 s, 40.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741392 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620655 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768254 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884165 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698536 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057314 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628645 s, 16.7 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg602-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg602-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3876 s, 10.1 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89933 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2903 s, 10.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90485 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (139s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 04:24:58 (1788596698) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg602-client.virtnet /mnt/lustre (opts:) Stopping client oleg602-client.virtnet /mnt/lustre opts: Stopping clients: oleg602-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg602-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg602-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg602-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg602-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg602-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg602-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.2,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg602-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.2,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg602-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.2,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69553/testresults/sanityn-zfs-DNE-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.2,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg602-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg602-server' oleg602-server: oleg602-server.virtnet: executing load_modules_local oleg602-server: kmemleak disabled oleg602-server: Loading modules from /home/green/git/lustre-release/lustre oleg602-server: detected 4 online CPUs by sysfs oleg602-server: MODOPTS_LIBCFS= oleg602-server: Force libcfs to create 2 CPU partitions oleg602-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg602-server: oleg602-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg602-client: oleg602-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg602-server: oleg602-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg602-client: oleg602-server: ssh exited with exit code 1 Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg602-server: oleg602-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg602-client: oleg602-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg602-server: oleg602-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg602-client: oleg602-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg602-client.virtnet: -o user_xattr,flock 192.168.206.102@tcp:/lustre /mnt/lustre Starting client oleg602-client.virtnet: -o user_xattr,flock 192.168.206.102@tcp:/lustre /mnt/lustre Started clients oleg602-client.virtnet: 192.168.206.102@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg602-client.virtnet: -o user_xattr,flock 192.168.206.102@tcp:/lustre /mnt/lustre2 Starting client oleg602-client.virtnet: -o user_xattr,flock 192.168.206.102@tcp:/lustre /mnt/lustre2 Started clients oleg602-client.virtnet: 192.168.206.102@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8b88902fb000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8b8893333800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b88902fb000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b8893333800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.496774 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586537 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456113 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869446 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105286 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1337 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125278 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112843 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143599 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29019 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315891 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302231 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299045 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336746 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341687 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342299 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.442722 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453457 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297759 s, 35.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062389 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711915 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398821 s, 26.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899887 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062156 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104676 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185143 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627131 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00871309 s, 120 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371807 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00601841 s, 174 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670054 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100238 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184139 s, 56.9 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg602-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg602-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.68816 s, 13.6 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.1189 s, 20.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.642201 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823435 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110293 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222812 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26116 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322894 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30118 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271845 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348463 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292665 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275313 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365429 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320321 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37193 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363329 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b8893333800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8b8893333800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359592 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.446144 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173543 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200151 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187018 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166356 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187306 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190647 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198071 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179472 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218891 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192889 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202561 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203048 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19819 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213184 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225514 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237203 s, 4.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b8893333800.lock_unused_count=2 removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg602-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg602-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.01642 s, 11.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94992 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (230s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 04:28:48 (1788596928) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg602-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (17s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 04:29:04 (1788596944) On MGS 192.168.206.102, active = nodemap.active=1 On MGS 192.168.206.102, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.793584 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494427 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479501 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510339 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444507 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140962 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259732 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205654 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256166 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269265 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251966 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210175 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242263 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309164 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316337 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397781 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33568 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330839 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362352 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29977 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349164 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304099 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347369 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372207 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34991 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336482 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358677 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373568 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364385 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406672 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371978 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357525 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.415557 s, 2.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b8893333800.lock_unused_count=5 removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.22232 s, 12.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.21245 s, 20.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.206.102, TBF.id = On MGS 192.168.206.102, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.674231 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474188 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113954 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923346 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852398 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249043 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122518 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08514 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270142 s, 3.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193637 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231079 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218144 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340078 s, 3.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b8893333800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362454 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384212 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347058 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.440364 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595596 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965966 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282213 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082161 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101293 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978595 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405848 s, 25.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043216 s, 24.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447573 s, 23.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477531 s, 22.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101819 s, 10.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104458 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117305 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525189 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560764 s, 18.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b8893333800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639647 s, 16.4 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg602-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg602-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.10304 s, 11.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.45798 s, 42.7 MB/s Read runtime is 3 s, speed is 33.333333 IOPS read rate (33.333333) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg602-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg602-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 On MGS 192.168.206.102, active = nodemap.active=1 On MGS 192.168.206.102, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.637834 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881847 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881775 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519609 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168814 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190695 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117533 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143723 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151907 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694209 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20381 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234066 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253353 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380853 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275961 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252163 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350573 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b8893333800.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925651 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227253 s, 46.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345333 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259215 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881835 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112973 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108479 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395063 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433722 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650288 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117914 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131018 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109205 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744248 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567471 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626197 s, 16.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8b8893333800.lock_unused_count=7 removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg602-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg602-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.07976 s, 11.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95369 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.206.102, TBF.id = On MGS 192.168.206.102, active = nodemap.active=0 PASS 77kj (185s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 04:32:09 (1788597129) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (10s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 04:32:20 (1788597140) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg602-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 4.09958 s, 256 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg602-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0045 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (59s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 04:33:19 (1788597199) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.467423 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663756 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082348 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124265 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113942 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120962 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105723 s, 9.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b8893333800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137374 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220597 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286567 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185034 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219315 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205002 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234255 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261858 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251368 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00358186 s, 293 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30604 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870926 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920126 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990898 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00586488 s, 179 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102422 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118366 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982432 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586612 s, 17.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603552 s, 17.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018269 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066667 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00365531 s, 287 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b8893333800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8b8893333800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00791955 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00312758 s, 335 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00916239 s, 114 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.39942 s, 12.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.25866 s, 19.9 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.572248 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587138 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056675 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788668 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121806 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204286 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214472 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228398 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187866 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189463 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227894 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178127 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195406 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237142 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8b8893333800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8b8893333800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243263 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258771 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20722 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.483887 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.521238 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.506655 s, 2.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.491881 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.496724 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.497164 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.501331 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.525511 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.539894 s, 1.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.488496 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.518828 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.50063 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.509685 s, 2.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.513696 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.520442 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.523333 s, 2.0 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg602-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg602-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.33572 s, 12.6 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.11802 s, 20.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (84s) == sanityn test 77o: Changing rank should not panic ====== 04:34:43 (1788597283) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (10s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 04:34:53 (1788597293) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (101s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 04:36:35 (1788597395) mds.MDS.mdt.nrs_policies=tbf oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg602-client: oleg602-server: ssh exited with exit code 36 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg602-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg602-client: oleg602-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (35s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 04:37:10 (1788597430) striped dir -i0 -c1 -H crush2 /mnt/lustre/d77r.sanityn mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 294717 kill md_thread xargs PASS 77r (56s) == sanityn test 77s: Check TBF LRU shrinker ============== 04:38:06 (1788597486) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 8 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 8 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (62s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 04:39:09 (1788597549) oleg602-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg602-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg602-client: oleg602-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (7s) == sanityn test 79: xattr: intent error ================== 04:39:16 (1788597556) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d79.sanityn File: /mnt/lustre/d79.sanityn Size: 32768 Blocks: 16 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129637319639057 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 04:39:18.000000000 -0400 Modify: 2026-09-05 04:39:18.000000000 -0400 Change: 2026-09-05 04:39:18.000000000 -0400 Birth: 2026-09-05 04:39:18.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (19s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 04:39:35 (1788597575) total: 10 open/close in 0.21 seconds: 48.67 ops/second multiop /mnt/lustre2/d80a.sanityn/f80a.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8428 Finish migration, then checking.. PASS 80a (11s) == sanityn test 80b: Accessing directory during migration ========================================================== 04:39:46 (1788597586) start migration thread 301832 accessing the migrating directory for 1 minute... sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete concurrent access 146 failures, 403 successes PASS 80b (86s) == sanityn test 81a: rename and stat under striped directory ========================================================== 04:41:12 (1788597672) stat: cannot statx '/mnt/lustre/d81a.sanityn/d0/0': No such file or directory File: /mnt/lustre/d81a.sanityn/d0/0 Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272381367867 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 04:41:13.000000000 -0400 Modify: 2026-09-05 04:41:13.000000000 -0400 Change: 2026-09-05 04:41:14.000000000 -0400 Birth: 2026-09-05 04:41:13.000000000 -0400 PASS 81a (5s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 04:41:17 (1788597677) - open/close 901 (time 1788597691.12 total 10.02 last 89.96) - open/close 1772 (time 1788597701.13 total 20.03 last 87.00) total: 2500 open/close in 28.13 seconds: 88.88 ops/second /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 304157 Killed ( while true; do touch $DIR1/$tdir; done ) (wd: ~) PASS 81b (430s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 04:48:28 (1788598108) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (2s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 04:48:30 (1788598110) - open/close 925 (time 1788598125.06 total 10.01 last 92.45) - open/close 1769 (time 1788598135.07 total 20.01 last 84.34) total: 2500 open/close in 27.79 seconds: 89.97 ops/second - open/close 837 (time 1788598158.20 total 10.01 last 83.61) - open/close 1626 (time 1788598168.20 total 20.01 last 78.88) - open/close 2462 (time 1788598178.21 total 30.02 last 83.57) total: 2500 open/close in 30.49 seconds: 82.00 ops/second snapshot_time 1788598318.474138794 secs.nsecs start_time 1788598112.786014617 secs.nsecs elapsed_time 205.688124177 secs.nsecs open 2505 samples [usecs] 116 14411 2965794 5878016792 close 2505 samples [usecs] 111 6440 875734 564062514 mknod 2501 samples [usecs] 481 11607 2475115 4167873887 unlink 2500 samples [usecs] 508 127925 6270981 88224641725 mkdir 2 samples [usecs] 1656 25032 26688 629343360 rmdir 2 samples [usecs] 634 8654 9288 75293672 rename 2500 samples [usecs] 410 60117 17370937 263156194997 getattr 3770 samples [usecs] 1 12095 244222 611559072 statfs 40 samples [usecs] 7 38 803 17245 parallel_rename_file 2500 samples [usecs] 410 60117 17370937 263156194997 crossdir_rename 2500 samples [usecs] 410 60117 17370937 263156194997 readpage 27 samples [usecs] 5417 83540 694510 25404590718 snapshot_time 1788598318.474441334 secs.nsecs start_time 1788598112.786079942 secs.nsecs elapsed_time 205.688361392 secs.nsecs open 2504 samples [usecs] 129 18623 3280665 8003838093 close 2504 samples [usecs] 134 9119 971180 890544988 mknod 2500 samples [usecs] 469 18246 2719552 5851763156 unlink 2500 samples [usecs] 528 78079 5787477 51566351509 mkdir 2 samples [usecs] 864 45164 46028 2040533392 rmdir 2 samples [usecs] 731 17988 18719 324102505 rename 2500 samples [usecs] 408 37095 12867638 131940351764 getattr 3758 samples [usecs] 10 26520 286322 1229180778 statfs 40 samples [usecs] 9 397 1110 172170 parallel_rename_file 2500 samples [usecs] 408 37095 12867638 131940351764 crossdir_rename 2500 samples [usecs] 408 37095 12867638 131940351764 readpage 20 samples [usecs] 6603 88602 484316 18368386542 PASS 81d (214s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 04:52:04 (1788598324) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.8428 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.8428 PASS 82 (8s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 04:52:12 (1788598332) start pid 318375 to create/unlink striped directory start pid 318376 to stat striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6198: 318375 Terminated ( cd $DIR1; while true; do $LFS mkdir -i1 -c2 $tdir; rmdir $tdir; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6198: 318376 Terminated ( cd $DIR2; while true; do stat $tdir > /dev/null 2>&1; done ) PASS 83 (128s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 04:54:20 (1788598460) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 04:54:33 (1788598473) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788598475 Finishing test test1 at 1788598480 PASS 85 (12s) == sanityn test 90: open/create and unlink striped directory ========================================================== 04:54:45 (1788598485) start pid 323211 to open/create under striped directory start pid 323213 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6265: 323211 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; touch $tdir/f{0..3} > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6265: 323213 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 90 (187s) == sanityn test 91: chmod and unlink striped directory === 04:57:52 (1788598672) start pid 325691 to chmod striped directory start pid 325692 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6302: 325691 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; chmod go+w $tdir > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6302: 325692 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 91 (187s) == sanityn test 92: create remote directory under orphan directory ========================================================== 05:00:59 (1788598859) fail_loc=0x1408 mkdir: cannot create directory '/mnt/lustre2/d92.sanityn/dir': No such file or directory lfs setdirstripe: cannot resolve path '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory (2) lfs setdirstripe: '/mnt/lustre2/d92.sanityn/remote_dir' is not on a Lustre filesystem: No such file or directory (2) lfs setdirstripe: cannot create dir '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory fail_loc=0 PASS 92 (8s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 05:01:07 (1788598867) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 3552 0xde0 0x2c0000400 0 3588 0xe04 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5449 0x1549 0x280000401 1 5415 0x1527 0x2c0000401 rc1=0 and rc2=0 PASS 93 (19s) == sanityn test 94: signal vs CP callback race =========== 05:01:26 (1788598886) osc.lustre-OST0000-osc-ffff8b88902fb000.idle_timeout=0 osc.lustre-OST0000-osc-ffff8b8893333800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8b88902fb000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8b8893333800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.218028 s, 37.6 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.95485 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 330602 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.177716 s, 23.0 kB/s osc.lustre-OST0000-osc-ffff8b88902fb000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8b8893333800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b88902fb000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b8893333800.idle_timeout=debug PASS 94 (16s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 05:01:42 (1788598902) llite.lustre-ffff8b88902fb000.fast_read=0 llite.lustre-ffff8b8893333800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0680308 s, 30.8 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8b88902fb000.fast_read=1 llite.lustre-ffff8b8893333800.fast_read=1 PASS 95a (20s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 05:02:03 (1788598923) llite.lustre-ffff8b88902fb000.fast_read=0 llite.lustre-ffff8b8893333800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0104138 s, 1.2 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8b88902fb000.fast_read=1 llite.lustre-ffff8b8893333800.fast_read=1 PASS 95b (15s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 05:02:17 (1788598937) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 05:02:21 (1788598941) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0119805 s, 342 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (8s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 05:02:29 (1788598949) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.207658 s, 10.1 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (7s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 05:02:36 (1788598956) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.197065 s, 10.6 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (7s) == sanityn test 100e: DoM: read on open and file size ==== 05:02:43 (1788598963) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00856887 s, 1.4 MB/s PASS 100e (7s) == sanityn test 101a: Discard DoM data on unlink ========= 05:02:50 (1788598970) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0075469 s, 543 kB/s PASS 101a (6s) == sanityn test 101b: Discard DoM data on rename ========= 05:02:57 (1788598977) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00936611 s, 437 kB/s PASS 101b (6s) == sanityn test 101c: Discard DoM data on close-unlink === 05:03:02 (1788598982) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00414656 s, 988 kB/s PASS 101c (6s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 05:03:08 (1788598988) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: a1 13 00 00 02 00 00 00 3a 18 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x2000013a1:0x183a:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.remote fh_bytes: 32 fh_type: 151 fh_data: d0 0b 00 40 02 00 00 00 d4 13 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000bd0:0x13d4:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.remote/subdir fh_bytes: 32 fh_type: 151 fh_data: a1 13 00 00 02 00 00 00 3b 18 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x2000013a1:0x183b:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.stripe fh_bytes: 32 fh_type: 151 fh_data: d0 0b 00 40 02 00 00 00 d5 13 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000bd0:0x13d5:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.stripe/subdir fh_bytes: 32 fh_type: 151 fh_data: d0 0b 00 40 02 00 00 00 d6 13 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000bd0:0x13d6:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! PASS 102 (9s) == sanityn test 103: Test size correctness with lockahead ========================================================== 05:03:17 (1788598997) striped dir -i1 -c2 -H crush /mnt/lustre/d103.sanityn fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788599000 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788599003 Finishing test test23 at 1788599003 PASS 103 (15s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 05:03:32 (1788599012) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (5s) == sanityn test 105: Glimpse and lock cancel race ======== 05:03:37 (1788599017) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d105.sanityn fail_loc=0x416 PASS 105 (23s) == sanityn test 106a: Verify the btime via statx() ======= 05:04:00 (1788599040) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 05:04:03 (1788599043) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0120371 s, 85.1 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381372478 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 05:04:04.000000000 -0400 Modify: 2026-09-05 05:04:04.000000000 -0400 Change: 2026-09-05 05:04:04.000000000 -0400 Birth: 2026-09-05 05:04:04.000000000 -0400 /mnt/lustre/f106b.sanityn 144115272381372478 -rw-r--r-- 1 0 0 1788599044 1788599044 1788599044 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381372478 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 05:04:04.000000000 -0400 Modify: 2026-09-05 05:04:04.000000000 -0400 Change: 2026-09-05 05:04:04.000000000 -0400 Birth: 2026-09-05 05:04:04.000000000 -0400 1788599044 1024 1 PASS 106b (7s) == sanityn test 106c: Verify statx attributes mask ======= 05:04:10 (1788599050) PASS 106c (6s) == sanityn test 107a: Basic grouplock conflict =========== 05:04:16 (1788599056) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.246858 s, 42.5 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5421 0x152d 0x2c0000401 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.8428 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.8428 First grouplock blocks second one PASS 107a (9s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 05:04:25 (1788599065) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.226013 s, 46.4 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5456 0x1550 0x280000401 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.8428 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (14s) == sanityn test 108a: lseek: parallel updates ============ 05:04:39 (1788599079) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00959685 s, 1.0 MB/s llite.lustre-ffff8b88902fb000.hybrid_io=0 llite.lustre-ffff8b8893333800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.96782 s, 1.7 MB/s PASS 108a (11s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 05:04:50 (1788599090) 192.168.206.102@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg602-client.virtnet /mnt/lustre (opts:) 192.168.206.102@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg602-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg602-client.virtnet: -o user_xattr,flock 192.168.206.102@tcp:/lustre /mnt/lustre2 Starting client: oleg602-client.virtnet: -o user_xattr,flock 192.168.206.102@tcp:/lustre /mnt/lustre 192.168.206.102@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.206.102@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg602-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg602-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg602-server' oleg602-server: oleg602-server.virtnet: executing load_modules_local oleg602-server: kmemleak disabled oleg602-server: Loading modules from /home/green/git/lustre-release/lustre oleg602-server: detected 4 online CPUs by sysfs oleg602-server: MODOPTS_LIBCFS= oleg602-server: Force libcfs to create 2 CPU partitions oleg602-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg602-client.virtnet: -o user_xattr,flock 192.168.206.102@tcp:/lustre /mnt/lustre Starting client: oleg602-client.virtnet: -o user_xattr,flock 192.168.206.102@tcp:/lustre /mnt/lustre2 192.168.206.102@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.206.102@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg602-client.virtnet /mnt/lustre (opts:) Stopping client oleg602-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg602-server' oleg602-server: oleg602-server.virtnet: executing load_modules_local oleg602-server: kmemleak disabled oleg602-server: Loading modules from /home/green/git/lustre-release/lustre oleg602-server: detected 4 online CPUs by sysfs oleg602-server: MODOPTS_LIBCFS= oleg602-server: Force libcfs to create 2 CPU partitions oleg602-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg602-client.virtnet: -o user_xattr,flock 192.168.206.102@tcp:/lustre /mnt/lustre Starting client: oleg602-client.virtnet: -o user_xattr,flock 192.168.206.102@tcp:/lustre /mnt/lustre2 PASS 109 (62s) == sanityn test 110: do not grant another lock on resend ========================================================== 05:05:53 (1788599153) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272431697923 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 05:05:54.000000000 -0400 Modify: 2026-09-05 05:05:54.000000000 -0400 Change: 2026-09-05 05:05:54.000000000 -0400 Birth: 2026-09-05 05:05:54.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg602-client.virtnet oleg602-client.virtnet PASS 110 (69s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 05:07:02 (1788599222) fail_loc=0x8000018a /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 7188: 346197 Terminated $MULTIOP $DIR2/$tdir/mdt0dir/foodir/file2 Ow4096_c PASS 111 (10s) == sanityn test 112: update max-inherit in default LMV === 05:07:12 (1788599232) lmv.lustre-clilmv-ffff8b8882e6e000.qos_threshold_rr=90 lmv.lustre-clilmv-ffff8b8886419000.qos_threshold_rr=90 PASS 112 (13s) == sanityn test 113: check servers of specified fs ======= 05:07:25 (1788599245) lustre-OST0000-osc-ffff8b8882e6e000 active. lustre-OST0001-osc-ffff8b8882e6e000 active. lustre-OST0000-osc-ffff8b8882e6e000 active. lustre-OST0001-osc-ffff8b8882e6e000 active. MGC192.168.206.102@tcp active. PASS 113 (6s) == sanityn test 114: implicit default LMV inherit ======== 05:07:31 (1788599251) /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 PASS 114 (21s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 05:07:53 (1788599273) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (2s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 05:07:55 (1788599275) llite.lustre-ffff8b8882e6e000.intent_mkdir=1 llite.lustre-ffff8b8886419000.intent_mkdir=1 MD layout /mnt/lustre/d116.sanityn: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x2000013a0:0x10e:0x0] 1 [0x240000bd2:0xfd:0x0] mkdir /mnt/lustre/d116.sanityn/tdir0 setdirstripe -D -i1 /mnt/lustre2/d116.sanityn/tdir0 mkdir /mnt/lustre/d116.sanityn/tdir0/tdir11 lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none PASS 116 (7s) == sanityn test 121: trunc append race =================== 05:08:02 (1788599282) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00968233 s, 2.1 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.1017 s, 201 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (10s) == sanityn test 122: directory size is consistent across mounts ========================================================== 05:08:12 (1788599292) PASS 122 (19s) == sanityn test 200: service remains healthy while able to process request ========================================================== 05:08:31 (1788599311) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d200.sanityn fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1343 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1051 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1585 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1532 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1403 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1328 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1594 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0481 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1505 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1804 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1424 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2104 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2529 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2036 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2682 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3227 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3118 s, 0.1 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3023 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg602-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b8882e6e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b8882e6e000.ost_server_uuid in FULL state after 0 sec PASS 200 (81s) cleanup: ====================================================== == sanityn test complete, duration 11858 sec ============= 05:09:54 (1788599394) === sanityn: start cleanup 05:09:56 (1788599396) === Stopping clients: oleg602-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg602-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 05:14:46 (1788599686) ===