-----============= acceptance-small: sanityn ============----- Sun Sep 6 19:38:21 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg231-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 19:38:27 (1788737907) === Starting client oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre2 Mount client oleg231-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre2 Started clients oleg231-client.virtnet: 192.168.202.131@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg231-client.virtnet: executing check_config_client /mnt/lustre oleg231-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg231-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b9e10447000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9b9e20121800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b9e10447000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b9e20121800.idle_timeout=debug disable quota as required oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 19:38:38 (1788737918) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011598 s, 90.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7630] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 19:38:38 (1788737918) PASS 0 (3s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 19:38:41 (1788737921) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (1s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 19:38:42 (1788737922) -rw-r--r-- 1 root root 0 Sep 6 19:38 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (2s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 19:38:44 (1788737924) -rw-r--r-- 1 root root 0 Sep 6 19:38 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (2s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 19:38:46 (1788737926) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (2s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 19:38:48 (1788737928) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (2s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 19:38:50 (1788737930) total 1 -rwxrwxrwx 1 root root 0 Sep 6 19:38 f2a -rwxrwxrwx 1 root root 0 Sep 6 19:38 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 6 19:38 f2a -rwxrwxrwx 1 root root 0 Sep 6 19:38 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00258108 s, 198 kB/s PASS 2e (2s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 19:38:53 (1788737933) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (1s) == sanityn test 2g: check blocks update on sync write ==== 19:38:54 (1788737934) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0803607 s, 26.1 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (2s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 19:38:56 (1788737936) PASS 3 (2s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 19:38:58 (1788737938) PASS 4 (3s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 19:39:01 (1788737941) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (2s) == sanityn test 6: remove of open file on other node ============================================================================ 19:39:03 (1788737943) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (2s) == sanityn test 7: remove of open directory on other node ======================================================================= 19:39:05 (1788737945) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (2s) == sanityn test 8: remove of open special file on other node ==================================================================== 19:39:07 (1788737947) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (2s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 19:39:09 (1788737949) PASS 9a (2s) == sanityn test 9b: append to striped sparse file ======== 19:39:11 (1788737951) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159399 s, 65.8 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.00890018 s, 0.3 kB/s Data read (expecting 'foo'): foo PASS 9b (2s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 19:39:14 (1788737954) 1+0 records in 1+0 records out 1 byte copied, 0.00157822 s, 0.6 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00267343 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00278015 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00300719 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00345645 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00275583 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00256282 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00247969 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00238931 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00258337 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00268906 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00305088 s, 0.3 kB/s PASS 10a (2s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 19:39:16 (1788737956) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00364313 s, 843 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00761283 s, 538 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000243078 s, 12.6 MB/s PASS 10b (2s) == sanityn test 11: execution of file opened for write should return error ============================================================== 19:39:18 (1788737958) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (2s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 19:39:20 (1788737960) warning: '-runtime' deprecated, use '-t runtime' instead - link 2565 (time 1788737971.32 total 10.00 last 256.41) - link 4731 (time 1788737981.33 total 20.01 last 216.52) - link 6721 (time 1788737991.33 total 30.01 last 198.97) - link 8848 (time 1788738001.33 total 40.01 last 212.65) - link 10000 (time 1788738006.41 total 45.09 last 227.04) - link 12204 (time 1788738016.41 total 55.09 last 220.37) - link 14487 (time 1788738026.41 total 65.09 last 228.26) - link 16823 (time 1788738036.41 total 75.09 last 233.58) - link 19132 (time 1788738046.41 total 85.09 last 230.85) - link 20000 (time 1788738050.19 total 88.87 last 229.42) - link 22431 (time 1788738060.20 total 98.88 last 243.01) total: 22700 link in 100.00 seconds: 226.99 ops/second using seed 581499906 running for 100 seconds - stat 10000 (time 1788737969 ; total 7 ; last 7) - stat 20000 (time 1788737976 ; total 14 ; last 7) - stat 30000 (time 1788737984 ; total 22 ; last 8) - stat 40000 (time 1788737992 ; total 30 ; last 8) - stat 50000 (time 1788737999 ; total 37 ; last 7) - stat 60000 (time 1788738006 ; total 44 ; last 7) - stat 69533 (time 1788738017 ; total 55 ; last 11) - stat 70000 (time 1788738018 ; total 56 ; last 1) - stat 75506 (time 1788738029 ; total 67 ; last 11) - stat 80000 (time 1788738037 ; total 75 ; last 8) - stat 85298 (time 1788738048 ; total 86 ; last 11) - stat 90000 (time 1788738057 ; total 95 ; last 9) total: 93305 stats in 100 seconds: 933.049988 stats/second - unlinked 0 (time 1788738012 ; total 0 ; last 0) - unlinked 10000 (time 1788738068 ; total 56 ; last 56) - unlinked 20000 (time 1788738095 ; total 83 ; last 27) unlink(/mnt/lustre2/lockdir/lockfile22700) error: No such file or directory total: 22699 unlinks in 90 seconds: 252.211105 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21404) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21406) - No such process PASS 12 (144s) == sanityn test 13: test directory page revocation ======= 19:41:44 (1788738104) f13 f13 PASS 13 (2s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 19:41:46 (1788738106) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22588 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (2s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 19:41:48 (1788738108) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23167 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (2s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 19:41:50 (1788738110) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23744 Terminated $DIR1/$tdir/sleep 60 PASS 14b (2s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 19:41:52 (1788738112) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24325 Terminated $DIR1/$tdir/sleep 60 PASS 14c (2s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 19:41:54 (1788738114) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24907 Terminated $DIR1/$tdir/sleep 60 PASS 14d (3s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 19:41:57 (1788738117) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7527424 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7527424KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (29s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 19:42:26 (1788738146) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.102806 s, 81.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0745211 s, 113 MB/s Chance of close/open is 1 in 50 Seed set to 8148 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole 000003[1] 1788738148.454900 mapwrite 0x1e63c8 thru 0x1e9f0b (0x3b44 bytes) 000005[1] 1788738148.483697 mapread 0x081349 thru 0x081fb4 (0xc6c bytes) 000007[1] 1788738148.493663 read 0x0a6a3c thru 0x0abf8a (0x554f bytes) 000008[1] 1788738148.499623 punch from 0x018725 to 0x025b34 (0xd410 bytes) truncating to largest ever: 0x30610 000009[1] 1788738148.505048 trunc from 0x1e9f0c to 0x03060f (0x1b98fc bytes) 000010[1] 1788738148.514306 read 0x00d957 thru 0x0112c8 (0x3972 bytes) 000011[1] 1788738148.521856 mapwrite 0x02f9ea thru 0x03a2ad (0xa8c4 bytes) 000012[1] 1788738148.543857 mapread 0x0367a0 thru 0x03a2ad (0x3b0e bytes) 000013[1] 1788738148.549603 write 0x807ef7 thru 0x812f4e (0xb058 bytes) 000014[0] 1788738148.553302 mapread 0x1b27e9 thru 0x1c22d7 (0xfaef bytes) 000015[0] 1788738148.571484 mapwrite 0x90f849 thru 0x919d60 (0xa518 bytes) 000017[0] 1788738148.606261 mapwrite 0x7a9ea5 thru 0x7afe47 (0x5fa3 bytes) 000018[1] 1788738148.625499 write 0x718167 thru 0x725954 (0xd7ee bytes) 000020[1] 1788738148.639049 mapread 0x51e5bc thru 0x52858b (0x9fd0 bytes) 000021[0] 1788738148.658399 write 0x99f105 thru 0x9a3169 (0x4065 bytes) truncating to largest ever: 0x3b2b17 000023[0] 1788738148.666656 trunc from 0x9a316a to 0x3b2b16 (0x5f0653 bytes) truncating to largest ever: 0x6a1065 000024[0] 1788738148.680886 trunc from 0x3b2b17 to 0x6a1064 (0x2ee54e bytes) 000026[1] 1788738148.692887 write 0x877b9d thru 0x881809 (0x9c6d bytes) 000027[0] 1788738148.699774 mapread 0x107b23 thru 0x11007e (0x855c bytes) 000030[0] 1788738148.711326 trunc from 0x88180a to 0x1ad978 (0x6d3e91 bytes) 000032[1] 1788738148.723671 mapread 0x1906e8 thru 0x191837 (0x1150 bytes) truncating to largest ever: 0x8cbe1d 000033[1] 1788738148.741725 trunc from 0x1ad979 to 0x8cbe1c (0x71e4a4 bytes) 000035[0] 1788738148.759729 mapwrite 0x8f0400 thru 0x9000b2 (0xfcb3 bytes) 000037[0] 1788738148.793779 write 0x8090a4 thru 0x80b401 (0x235e bytes) 000038[1] 1788738148.796596 punch from 0x12c7b1 to 0x1314af (0x4cff bytes) 000041[1] 1788738148.811925 trunc from 0x9000b3 to 0x25391d (0x6ac795 bytes) 000045[0] 1788738148.826181 read 0x0535e1 thru 0x05aabf (0x74df bytes) 000046[0] 1788738148.832013 mapwrite 0x634258 thru 0x63cdf5 (0x8b9e bytes) 000048[0] 1788738148.865215 mapwrite 0x3c6126 thru 0x3d1431 (0xb30c bytes) 000049[1] 1788738148.892673 trunc from 0x63cdf6 to 0x0781cc (0x5c4c29 bytes) 000051[1] 1788738148.911278 punch from 0x05d71e to 0x05e502 (0xde5 bytes) 000054[0] 1788738148.923300 mapread 0x06f009 thru 0x071f12 (0x2f0a bytes) 000055[0] 1788738148.934023 trunc from 0x0781cd to 0x5a10bb (0x528eef bytes) 000057[0] 1788738148.946620 read 0x14a7ff thru 0x14bd7f (0x1581 bytes) 000058[0] 1788738148.953673 trunc from 0x5a10bc to 0x533a85 (0x6d636 bytes) 000059[1] 1788738148.963124 punch from 0x008e19 to 0x013f8f (0xb177 bytes) 000060[0] 1788738148.970601 mapread 0x358a6a thru 0x359393 (0x92a bytes) 000062[0] 1788738148.983225 mapread 0x10ebb8 thru 0x11007c (0x14c5 bytes) 000065[1] 1788738148.992727 read 0x44bf83 thru 0x44d0bb (0x1139 bytes) 000068[1] 1788738148.998447 mapread 0x1c850f thru 0x1cdc6e (0x5760 bytes) 000073[1] 1788738149.004658 read 0x308d63 thru 0x30dfa0 (0x523e bytes) 000074[0] 1788738149.007047 write 0x28a49e thru 0x294faf (0xab12 bytes) 000075[0] 1788738149.015414 mapread 0x1c9831 thru 0x1ccf34 (0x3704 bytes) 000078[0] 1788738149.026996 read 0x0af33a thru 0x0b8f65 (0x9c2c bytes) 000080[0] 1788738149.030129 write 0x03bc1d thru 0x0441ea (0x85ce bytes) 000083[0] 1788738149.037824 write 0x99bce4 thru 0x9a0dc3 (0x50e0 bytes) 000087[0] 1788738149.040320 trunc from 0x9a0dc4 to 0x4ef4ce (0x4b18f5 bytes) 000090[1] 1788738149.055105 punch from 0x128beb to 0x13454e (0xb964 bytes) 000091[1] 1788738149.065128 mapread 0x029b3c thru 0x039880 (0xfd45 bytes) 000093[0] 1788738149.081583 mapwrite 0x1fc298 thru 0x1fce2c (0xb95 bytes) 000095[1] 1788738149.103190 mapwrite 0x5ef617 thru 0x5fe84a (0xf234 bytes) 000096[1] 1788738149.132289 trunc from 0x5fe84b to 0x35e297 (0x2a05b3 bytes) 000097[1] 1788738149.136215 punch from 0x09982c to 0x0a5f97 (0xc76c bytes) 000098[0] 1788738149.137523 write 0x1f6ccd thru 0x205c33 (0xef67 bytes) 000099[1] 1788738149.146025 mapwrite 0x171843 thru 0x17ca20 (0xb1de bytes) 000101[1] 1788738149.182659 write 0x21c30b thru 0x21f54a (0x3240 bytes) 000102[0] 1788738149.185016 punch from 0x1fe0ed to 0x1ff841 (0x1755 bytes) 000103[0] 1788738149.194265 mapwrite 0x1051aa thru 0x105667 (0x4be bytes) 000104[0] 1788738149.213127 mapwrite 0x2ba04e thru 0x2c8f75 (0xef28 bytes) 000105[1] 1788738149.242063 trunc from 0x35e298 to 0x5894cc (0x22b235 bytes) 000106[1] 1788738149.250439 mapwrite 0x1cde2d thru 0x1d8dcd (0xafa1 bytes) 000107[0] 1788738149.272937 mapread 0x55aafe thru 0x55ef0f (0x4412 bytes) 000109[1] 1788738149.289565 mapwrite 0x583c51 thru 0x58e763 (0xab13 bytes) 000113[1] 1788738149.332713 mapread 0x37f76e thru 0x38eb9d (0xf430 bytes) 000116[1] 1788738149.354660 mapwrite 0x4f3dbd thru 0x501b27 (0xdd6b bytes) 000117[1] 1788738149.379826 mapread 0x24e151 thru 0x251ec9 (0x3d79 bytes) 000118[0] 1788738149.386089 trunc from 0x58e764 to 0x188866 (0x405efd bytes) 000119[1] 1788738149.398362 mapread 0x0096c9 thru 0x00f368 (0x5ca0 bytes) 000120[0] 1788738149.412898 punch from 0x108f19 to 0x11785a (0xe942 bytes) 000123[0] 1788738149.429481 mapread 0x13491f thru 0x142224 (0xd906 bytes) 000124[0] 1788738149.436861 punch from 0x0dd89c to 0x0e1459 (0x3bbe bytes) 000125[1] 1788738149.438489 punch from 0x0e5f42 to 0x0f001a (0xa0d9 bytes) 000126[0] 1788738149.445909 punch from 0x13fe08 to 0x14c837 (0xca30 bytes) 000128[0] 1788738149.454676 read 0x1763e7 thru 0x1843cf (0xdfe9 bytes) 000129[0] 1788738149.470820 trunc from 0x188867 to 0x3a196b (0x219105 bytes) 000132[0] 1788738149.478038 punch from 0x172723 to 0x1731cd (0xaab bytes) 000134[0] 1788738149.480124 punch from 0x0f3016 to 0x0feb01 (0xbaec bytes) 000135[0] 1788738149.481978 mapwrite 0x2241c0 thru 0x22bddf (0x7c20 bytes) 000137[1] 1788738149.512335 write 0x1fddd9 thru 0x20072a (0x2952 bytes) 000138[0] 1788738149.524887 punch from 0x2b5dfa to 0x2c2619 (0xc820 bytes) 000139[0] 1788738149.530259 mapwrite 0x2306ce thru 0x23d065 (0xc998 bytes) 000140[0] 1788738149.566357 mapwrite 0x7e6dc2 thru 0x7f45cb (0xd80a bytes) 000143[1] 1788738149.622226 trunc from 0x7f45cc to 0x42e3b0 (0x3c621b bytes) 000145[0] 1788738149.646273 write 0x1bca83 thru 0x1c8b50 (0xc0ce bytes) 000146[1] 1788738149.655934 write 0x584e17 thru 0x5944d4 (0xf6be bytes) truncating to largest ever: 0x9dd158 000148[0] 1788738149.665244 trunc from 0x5944d5 to 0x9dd157 (0x448c83 bytes) 000149[0] 1788738149.677448 write 0x165351 thru 0x17027c (0xaf2c bytes) 000150[0] 1788738149.686163 trunc from 0x9dd158 to 0x9824d7 (0x5ac80 bytes) 000151[1] 1788738149.691634 mapread 0x5d3463 thru 0x5d3b5e (0x6fc bytes) 000152[0] 1788738149.702445 mapwrite 0x996e03 thru 0x9a6446 (0xf644 bytes) 000153[0] 1788738149.743504 write 0x402802 thru 0x407c4c (0x544b bytes) 000158[1] 1788738149.760246 read 0x3717bc thru 0x37641f (0x4c64 bytes) 000159[1] 1788738149.767557 write 0x92d171 thru 0x93730e (0xa19e bytes) 000160[0] 1788738149.777090 write 0x59b3b6 thru 0x59f0f5 (0x3d40 bytes) 000164[0] 1788738149.796811 punch from 0x79ca98 to 0x7aada5 (0xe30e bytes) 000167[1] 1788738149.810004 write 0x24c8fd thru 0x25332e (0x6a32 bytes) 000168[0] 1788738149.816936 mapread 0x67d2a8 thru 0x67d8a7 (0x600 bytes) 000169[1] 1788738149.824276 trunc from 0x9a6447 to 0x107889 (0x89ebbd bytes) 000170[0] 1788738149.835978 mapread 0x0d9569 thru 0x0e25ce (0x9066 bytes) 000172[0] 1788738149.859381 trunc from 0x10788a to 0x513e82 (0x40c5f9 bytes) 000173[1] 1788738149.870904 mapread 0x4307f0 thru 0x437efe (0x770f bytes) 000175[1] 1788738149.882800 read 0x3589fa thru 0x3619c9 (0x8fd0 bytes) 000176[1] 1788738149.889418 trunc from 0x513e83 to 0x4f0bbf (0x232c3 bytes) 000177[1] 1788738149.900856 mapread 0x40f866 thru 0x41b187 (0xb922 bytes) 000178[1] 1788738149.914592 read 0x4464c7 thru 0x4503fd (0x9f37 bytes) 000179[1] 1788738149.921321 mapwrite 0x9c38b6 thru 0x9c7dd2 (0x451d bytes) 000180[1] 1788738149.947588 read 0x8e1ac7 thru 0x8eccd1 (0xb20b bytes) 000181[0] 1788738149.950399 mapwrite 0x41c8c7 thru 0x42be69 (0xf5a3 bytes) 000182[0] 1788738149.995295 write 0x649ab0 thru 0x64a773 (0xcc4 bytes) 000183[0] 1788738150.006622 trunc from 0x9c7dd3 to 0x8615ee (0x1667e4 bytes) 000184[0] 1788738150.020191 mapwrite 0x0aabdf thru 0x0b15f6 (0x6a18 bytes) 000186[0] 1788738150.052737 mapwrite 0x39db98 thru 0x3a6982 (0x8deb bytes) 000189[0] 1788738150.088291 write 0x9f262e thru 0x9f372e (0x1101 bytes) 000190[1] 1788738150.089609 mapread 0x3cc4fc thru 0x3d4ba8 (0x86ad bytes) 000193[1] 1788738150.111426 trunc from 0x9f372f to 0x69be1f (0x35790f bytes) 000194[1] 1788738150.122215 mapread 0x3393b6 thru 0x3475d5 (0xe220 bytes) 000197[0] 1788738150.140653 punch from 0x2e2490 to 0x2f100e (0xeb7f bytes) 000200[1] 1788738150.155488 read 0x0e1168 thru 0x0e4511 (0x33aa bytes) 000202[1] 1788738150.163867 mapread 0x2c50de thru 0x2d4e73 (0xfd96 bytes) 000203[0] 1788738150.175623 mapread 0x2b1c7f thru 0x2b384b (0x1bcd bytes) 000204[0] 1788738150.184667 read 0x43de94 thru 0x444302 (0x646f bytes) 000205[0] 1788738150.189442 trunc from 0x69be20 to 0x9b3fe8 (0x3181c9 bytes) 000206[1] 1788738150.205281 read 0x1cedad thru 0x1d0abe (0x1d12 bytes) 000208[0] 1788738150.215118 read 0x37081d thru 0x37aec3 (0xa6a7 bytes) 000209[0] 1788738150.217656 trunc from 0x9b3fe9 to 0x0d67fb (0x8dd7ed bytes) 000211[1] 1788738150.230924 mapwrite 0x1a452c thru 0x1af481 (0xaf56 bytes) 000214[1] 1788738150.267085 mapwrite 0x52ba92 thru 0x5315c8 (0x5b37 bytes) 000216[0] 1788738150.301368 punch from 0x2742dc to 0x275269 (0xf8e bytes) 000217[1] 1788738150.309117 punch from 0x13ba08 to 0x141e63 (0x645c bytes) 000218[0] 1788738150.317843 punch from 0x00eefe to 0x01214c (0x324f bytes) 000219[0] 1788738150.325560 trunc from 0x5315c9 to 0x9900e2 (0x45eb1a bytes) 000220[0] 1788738150.334963 trunc from 0x9900e3 to 0x06826e (0x927e74 bytes) 000221[1] 1788738150.339377 punch from 0x05f2ff to 0x0669bb (0x76bd bytes) 000224[1] 1788738150.356138 mapread 0x065e03 thru 0x06826e (0x246c bytes) 000226[1] 1788738150.370525 mapwrite 0x78790f thru 0x796ee5 (0xf5d7 bytes) 000227[1] 1788738150.404237 punch from 0x621720 to 0x6246a1 (0x2f82 bytes) 000228[0] 1788738150.411233 punch from 0x71cc23 to 0x724738 (0x7b16 bytes) 000232[0] 1788738150.429994 write 0x021369 thru 0x02724e (0x5ee6 bytes) 000233[0] 1788738150.439811 trunc from 0x796ee6 to 0x0bff17 (0x6d6fce bytes) 000235[1] 1788738150.445808 mapread 0x0af77f thru 0x0b8240 (0x8ac2 bytes) 000236[1] 1788738150.458567 trunc from 0x0bff18 to 0x791403 (0x6d14ec bytes) 000238[0] 1788738150.471329 mapread 0x080cda thru 0x085d40 (0x5067 bytes) 000239[0] 1788738150.485981 write 0x726cae thru 0x730fc0 (0xa313 bytes) 000242[1] 1788738150.499391 mapwrite 0x3bec52 thru 0x3c837b (0x972a bytes) 000245[1] 1788738150.535277 punch from 0x2ae78f to 0x2b749c (0x8d0e bytes) 000246[1] 1788738150.540830 mapread 0x6ff46f thru 0x70a533 (0xb0c5 bytes) 000247[1] 1788738150.557422 punch from 0x32df52 to 0x33d1b8 (0xf267 bytes) 000248[0] 1788738150.562227 punch from 0x50940e to 0x518b10 (0xf703 bytes) 000249[0] 1788738150.569632 punch from 0x41abc4 to 0x420637 (0x5a74 bytes) 000250[1] 1788738150.574689 mapwrite 0x3c66a8 thru 0x3c9284 (0x2bdd bytes) 000251[1] 1788738150.592825 punch from 0x0a0ae9 to 0x0af24c (0xe764 bytes) 000252[0] 1788738150.597583 write 0x121abf thru 0x12df91 (0xc4d3 bytes) 000255[0] 1788738150.610984 read 0x50157d thru 0x506f94 (0x5a18 bytes) 000256[0] 1788738150.617509 trunc from 0x791404 to 0x10b00e (0x6863f5 bytes) 000257[1] 1788738150.626693 trunc from 0x10b00f to 0x060f02 (0xaa10c bytes) 000258[0] 1788738150.638273 trunc from 0x060f03 to 0x6e9c60 (0x688d5e bytes) 000260[0] 1788738150.654554 punch from 0x2f41ed to 0x302f3a (0xed4e bytes) 000262[0] 1788738150.660377 read 0x656890 thru 0x65b286 (0x49f7 bytes) 000264[1] 1788738150.667442 punch from 0x5b360c to 0x5be216 (0xac0b bytes) 000265[0] 1788738150.674387 trunc from 0x6e9c61 to 0x74bcb4 (0x62054 bytes) 000266[0] 1788738150.682013 mapwrite 0x4b0f56 thru 0x4b71bf (0x626a bytes) 000269[1] 1788738150.706838 trunc from 0x74bcb5 to 0x3e532f (0x366985 bytes) 000270[0] 1788738150.720259 write 0x0d31fa thru 0x0d3e63 (0xc6a bytes) 000271[0] 1788738150.728177 mapwrite 0x630459 thru 0x63b4ba (0xb062 bytes) 000273[0] 1788738150.761982 mapwrite 0x0c8064 thru 0x0d335b (0xb2f8 bytes) 000275[0] 1788738150.789992 trunc from 0x63b4bb to 0x614881 (0x26c39 bytes) 000279[0] 1788738150.805193 mapread 0x151100 thru 0x159435 (0x8336 bytes) 000281[1] 1788738150.808567 read 0x155ce9 thru 0x157e85 (0x219d bytes) 000282[0] 1788738150.813584 mapwrite 0x0ecd86 thru 0x0f6977 (0x9bf2 bytes) 000283[0] 1788738150.833488 trunc from 0x614882 to 0x27906d (0x39b814 bytes) 000284[1] 1788738150.843571 mapread 0x0e9d79 thru 0x0f2055 (0x82dd bytes) 000285[1] 1788738150.855015 punch from 0x08e78c to 0x09e06c (0xf8e1 bytes) 000289[1] 1788738150.873320 mapread 0x278de2 thru 0x27906d (0x28c bytes) 000290[0] 1788738150.883342 punch from 0x05ba70 to 0x0633d5 (0x7966 bytes) 000292[0] 1788738150.894750 trunc from 0x27906e to 0x6ee9b6 (0x475949 bytes) 000295[1] 1788738150.902189 mapwrite 0x49bde4 thru 0x4a28a2 (0x6abf bytes) 000296[1] 1788738150.928937 mapwrite 0x4b1779 thru 0x4bcfe2 (0xb86a bytes) 000299[0] 1788738150.959558 mapread 0x30bf4a thru 0x312b63 (0x6c1a bytes) 000300[0] 1788738150.972358 mapwrite 0x085bbb thru 0x08e05b (0x84a1 bytes) 000301[0] 1788738150.995205 write 0x71fc97 thru 0x72476a (0x4ad4 bytes) 000302[0] 1788738151.002643 mapread 0x6af18b thru 0x6b9ea5 (0xad1b bytes) 000304[0] 1788738151.011887 write 0x0d6d0b thru 0x0e5996 (0xec8c bytes) 000305[0] 1788738151.019617 write 0x0b831e thru 0x0c2033 (0x9d16 bytes) 000307[0] 1788738151.027624 mapread 0x443840 thru 0x448b77 (0x5338 bytes) 000309[0] 1788738151.037150 write 0x01e3d8 thru 0x021254 (0x2e7d bytes) 000310[0] 1788738151.039931 mapwrite 0x6408d6 thru 0x64b212 (0xa93d bytes) 000312[1] 1788738151.065628 trunc from 0x72476b to 0x0a3247 (0x681523 bytes) 000313[0] 1788738151.083094 read 0x0785ed thru 0x081a1e (0x9432 bytes) 000314[0] 1788738151.093685 mapread 0x093ce2 thru 0x0a3247 (0xf566 bytes) 000316[1] 1788738151.100570 mapwrite 0x108fc0 thru 0x1126b1 (0x96f2 bytes) 000320[1] 1788738151.136970 mapread 0x0c9909 thru 0x0d7f76 (0xe66e bytes) 000321[0] 1788738151.144977 trunc from 0x1126b2 to 0x61dd55 (0x50b6a4 bytes) 000322[1] 1788738151.156717 trunc from 0x61dd56 to 0x58b7ed (0x92568 bytes) 000323[1] 1788738151.166260 trunc from 0x58b7ee to 0x4dba69 (0xafd84 bytes) 000324[1] 1788738151.174711 mapwrite 0x94cab1 thru 0x95a6ec (0xdc3c bytes) 000325[1] 1788738151.201672 write 0x83f3a9 thru 0x8488c6 (0x951e bytes) 000326[0] 1788738151.205326 punch from 0x2eb0bc to 0x2f5629 (0xa56e bytes) 000327[0] 1788738151.216776 read 0x413f67 thru 0x415501 (0x159b bytes) 000329[0] 1788738151.225222 mapwrite 0x2f8352 thru 0x307a57 (0xf706 bytes) 000330[1] 1788738151.252177 read 0x477cfe thru 0x47ea81 (0x6d84 bytes) 000332[0] 1788738151.263381 punch from 0x269455 to 0x273b34 (0xa6e0 bytes) 000334[0] 1788738151.272085 trunc from 0x95a6ed to 0x7f2e45 (0x1678a7 bytes) 000335[0] 1788738151.280195 write 0x19c1ae thru 0x1aab24 (0xe977 bytes) 000336[1] 1788738151.283687 read 0x566753 thru 0x5706d5 (0x9f83 bytes) 000337[0] 1788738151.290017 write 0x0ac522 thru 0x0b565b (0x913a bytes) 000338[0] 1788738151.297690 punch from 0x6a1767 to 0x6a7f7e (0x6818 bytes) 000340[0] 1788738151.311106 trunc from 0x7f2e46 to 0x188138 (0x66ad0d bytes) 000343[1] 1788738151.325011 write 0x611507 thru 0x620c37 (0xf731 bytes) 000344[1] 1788738151.334074 punch from 0x1adaf2 to 0x1b197a (0x3e89 bytes) 000345[0] 1788738151.339990 mapwrite 0x600b1f thru 0x60a27d (0x975f bytes) 000346[1] 1788738151.369829 punch from 0x206226 to 0x206440 (0x21b bytes) 000348[0] 1788738151.378645 write 0x50ca47 thru 0x50e411 (0x19cb bytes) 000349[1] 1788738151.385413 punch from 0x3dba7e to 0x3dced7 (0x145a bytes) 000350[0] 1788738151.390661 read 0x317871 thru 0x31c2d5 (0x4a65 bytes) 000352[0] 1788738151.400761 mapwrite 0x052810 thru 0x060764 (0xdf55 bytes) 000356[0] 1788738151.426695 punch from 0x527361 to 0x52753e (0x1de bytes) 000358[1] 1788738151.434353 mapread 0x0e012f thru 0x0eb66c (0xb53e bytes) 000359[0] 1788738151.448974 read 0x590212 thru 0x59df37 (0xdd26 bytes) 000360[0] 1788738151.452851 punch from 0x478db4 to 0x47b9eb (0x2c38 bytes) 000361[0] 1788738151.457843 trunc from 0x620c38 to 0x35a570 (0x2c66c7 bytes) 000363[0] 1788738151.466234 punch from 0x15ffe6 to 0x168f84 (0x8f9f bytes) 000364[0] 1788738151.467957 write 0x2684e2 thru 0x275dfc (0xd91b bytes) 000365[0] 1788738151.476337 mapwrite 0x8bebd7 thru 0x8bf7b0 (0xbda bytes) 000366[0] 1788738151.499119 mapread 0x5a2122 thru 0x5a3200 (0x10df bytes) 000367[0] 1788738151.502171 mapread 0x270e63 thru 0x27c54f (0xb6ed bytes) 000368[1] 1788738151.506921 mapread 0x127cc3 thru 0x135b8b (0xdec9 bytes) 000370[0] 1788738151.521929 punch from 0x03a216 to 0x03ea85 (0x4870 bytes) 000371[1] 1788738151.525990 read 0x4fda73 thru 0x503cd1 (0x625f bytes) 000373[1] 1788738151.534955 trunc from 0x8bf7b1 to 0x3f3188 (0x4cc628 bytes) 000374[0] 1788738151.544137 mapread 0x13a43d thru 0x143be7 (0x97ab bytes) 000376[1] 1788738151.558637 write 0x3ad9d8 thru 0x3bb4b3 (0xdadc bytes) 000377[0] 1788738151.564115 trunc from 0x3f3189 to 0x0c559c (0x32dbec bytes) 000380[0] 1788738151.586556 write 0x131f4c thru 0x13dbee (0xbca3 bytes) 000382[1] 1788738151.588047 punch from 0x08135a to 0x085a96 (0x473d bytes) 000383[1] 1788738151.597620 read 0x11bf5a thru 0x12affe (0xf0a5 bytes) 000384[0] 1788738151.609893 write 0x4bcafb thru 0x4bcf1d (0x423 bytes) 000385[0] 1788738151.615960 punch from 0x303a6b to 0x311d3a (0xe2d0 bytes) 000390[1] 1788738151.622503 write 0x1473f3 thru 0x1564f8 (0xf106 bytes) 000391[1] 1788738151.629628 punch from 0x0716b7 to 0x080030 (0xe97a bytes) 000392[0] 1788738151.635641 punch from 0x05c58f to 0x0644fb (0x7f6d bytes) 000394[1] 1788738151.646059 trunc from 0x4bcf1e to 0x5209fb (0x63ade bytes) 000395[0] 1788738151.656976 read 0x2ce710 thru 0x2d86ca (0x9fbb bytes) 000396[0] 1788738151.666725 write 0x90e933 thru 0x90fc9a (0x1368 bytes) 000397[0] 1788738151.674838 read 0x524c95 thru 0x53158f (0xc8fb bytes) 000398[0] 1788738151.692093 punch from 0x64edb3 to 0x65b13c (0xc38a bytes) 000399[0] 1788738151.700568 mapwrite 0x808017 thru 0x81251e (0xa508 bytes) 000401[0] 1788738151.724703 punch from 0x6ed292 to 0x6fcf3b (0xfcaa bytes) 000402[0] 1788738151.726237 punch from 0x5534d9 to 0x559584 (0x60ac bytes) 000403[1] 1788738151.731998 mapwrite 0x1acd53 thru 0x1b4f0f (0x81bd bytes) 000408[0] 1788738151.781385 mapread 0x71a40d thru 0x71aa44 (0x638 bytes) 000415[0] 1788738151.811749 punch from 0x4f56bd to 0x4fe4be (0x8e02 bytes) 000416[1] 1788738151.817050 punch from 0x3d640d to 0x3e321c (0xce10 bytes) 000418[0] 1788738151.825540 read 0x0889b4 thru 0x08f979 (0x6fc6 bytes) 000419[1] 1788738151.836846 write 0x8aaf90 thru 0x8b1043 (0x60b4 bytes) 000428[0] 1788738151.881100 write 0x54c930 thru 0x558d0a (0xc3db bytes) 000430[1] 1788738151.890152 write 0x41d8f3 thru 0x428182 (0xa890 bytes) 000431[1] 1788738151.909062 write 0x0f3221 thru 0x0f8f44 (0x5d24 bytes) 000432[0] 1788738151.912214 write 0x42f088 thru 0x42fc5b (0xbd4 bytes) 000433[1] 1788738151.923646 mapwrite 0x4bcaf8 thru 0x4c7266 (0xa76f bytes) 000434[0] 1788738151.963217 trunc from 0x90fc9b to 0x89ded4 (0x71dc6 bytes) 000435[1] 1788738151.977224 trunc from 0x89ded5 to 0x1b4394 (0x6e9b40 bytes) 000437[1] 1788738151.996850 punch from 0x18b9ff to 0x1938de (0x7ee0 bytes) 000438[1] 1788738152.003523 mapread 0x09ff1d thru 0x0aafd2 (0xb0b6 bytes) 000439[0] 1788738152.024043 read 0x06d261 thru 0x07bce7 (0xea87 bytes) 000441[0] 1788738152.037985 mapwrite 0x329ab0 thru 0x336c0f (0xd160 bytes) 000442[1] 1788738152.079199 punch from 0x3267d7 to 0x331f64 (0xb78e bytes) 000443[0] 1788738152.089168 trunc from 0x336c10 to 0x90277c (0x5cbb6d bytes) 000444[1] 1788738152.102428 punch from 0x4f38e1 to 0x4f6c80 (0x33a0 bytes) 000451[0] 1788738152.133674 punch from 0x530947 to 0x5315e2 (0xc9c bytes) 000454[1] 1788738152.146163 punch from 0x2fe1a3 to 0x3011e6 (0x3044 bytes) 000455[1] 1788738152.152981 trunc from 0x90277d to 0x433795 (0x4cefe7 bytes) 000456[1] 1788738152.160538 read 0x036c18 thru 0x0380cc (0x14b5 bytes) 000457[1] 1788738152.162340 write 0x6bb7f2 thru 0x6c6505 (0xad14 bytes) 000458[1] 1788738152.169355 trunc from 0x6c6506 to 0x9c27c5 (0x2fc2c0 bytes) 000459[1] 1788738152.179658 trunc from 0x9c27c6 to 0x5c3541 (0x3ff284 bytes) 000462[1] 1788738152.193163 punch from 0x45fbf2 to 0x46d1f6 (0xd605 bytes) 000466[1] 1788738152.210676 read 0x0e799d thru 0x0f2fce (0xb632 bytes) 000468[1] 1788738152.222423 mapwrite 0x32108d thru 0x32b146 (0xa0ba bytes) 000470[1] 1788738152.246655 punch from 0x2d7fe3 to 0x2dc3db (0x43f9 bytes) 000471[0] 1788738152.253379 write 0x1bc187 thru 0x1cbd26 (0xfba0 bytes) 000472[0] 1788738152.268289 mapwrite 0x167d01 thru 0x1742ef (0xc5ef bytes) 000473[1] 1788738152.307413 write 0x6c98ef thru 0x6cec77 (0x5389 bytes) 000474[0] 1788738152.311965 read 0x1021e5 thru 0x104a3a (0x2856 bytes) 000475[0] 1788738152.317236 mapwrite 0x63c03d thru 0x646fc7 (0xaf8b bytes) 000477[1] 1788738152.346564 mapwrite 0x88656f thru 0x8963f4 (0xfe86 bytes) 000478[0] 1788738152.382336 trunc from 0x8963f5 to 0x79ea8f (0xf7965 bytes) 000480[0] 1788738152.394983 write 0x791d96 thru 0x795714 (0x397f bytes) 000481[1] 1788738152.401785 read 0x236201 thru 0x2443cf (0xe1cf bytes) 000482[1] 1788738152.407339 trunc from 0x79ea90 to 0x68c5c0 (0x1124cf bytes) 000483[1] 1788738152.423193 read 0x5ad94d thru 0x5b3fbf (0x6673 bytes) 000484[1] 1788738152.430084 punch from 0x5afd6b to 0x5b6c92 (0x6f28 bytes) 000485[1] 1788738152.435850 write 0x35079f thru 0x36006a (0xf8cc bytes) 000486[0] 1788738152.440415 mapread 0x3c71cd thru 0x3caef6 (0x3d2a bytes) 000489[1] 1788738152.458017 write 0x45dae3 thru 0x46ad10 (0xd22e bytes) 000490[1] 1788738152.465008 punch from 0x0dde40 to 0x0e7f65 (0xa126 bytes) 000492[1] 1788738152.469215 read 0x4f1631 thru 0x4f53a5 (0x3d75 bytes) 000494[0] 1788738152.479803 write 0x59bdeb thru 0x5abb51 (0xfd67 bytes) 000496[0] 1788738152.494405 write 0x3b880a thru 0x3c286e (0xa065 bytes) 000497[0] 1788738152.499708 read 0x4344d8 thru 0x43f8ec (0xb415 bytes) 000498[1] 1788738152.502291 mapread 0x281052 thru 0x290faa (0xff59 bytes) 000499[1] 1788738152.518823 read 0x2a1052 thru 0x2ae498 (0xd447 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 8152 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1d9f39 000003[1] 1788738152.813696 trunc from 00000000 to 0x1d9f38 (0x1d9f39 bytes) 000004[0] 1788738152.825490 read 0x077000 thru 0x078877 (0x1878 bytes) 000005[0] 1788738152.833739 mapread 0x15c000 thru 0x1638ee (0x78ef bytes) 000006[0] 1788738152.839094 read 0x17b000 thru 0x17c3df (0x13e0 bytes) 000009[0] 1788738152.854646 write 0x8fc000 thru 0x8fdfb0 (0x1fb1 bytes) 000011[1] 1788738152.866231 mapwrite 0x33c000 thru 0x347d27 (0xbd28 bytes) 000012[0] 1788738152.898475 read 0x23b000 thru 0x23bc29 (0xc2a bytes) 000013[0] 1788738152.903070 mapwrite 0x5fb000 thru 0x60555e (0xa55f bytes) truncating to largest ever: 0x72c090 000015[0] 1788738152.929877 trunc from 0x8fdfb1 to 0x72c08f (0x1d1f21 bytes) 000016[0] 1788738152.935549 read 0x02a000 thru 0x02ed7f (0x4d80 bytes) 000019[0] 1788738152.946300 write 0x959000 thru 0x95a337 (0x1338 bytes) 000021[1] 1788738152.951268 read 0x64c000 thru 0x65aba5 (0xeba6 bytes) 000023[1] 1788738152.958832 mapwrite 0x0e5000 thru 0x0f0c1c (0xbc1d bytes) 000024[0] 1788738152.982746 mapwrite 0x967000 thru 0x969f81 (0x2f82 bytes) 000025[1] 1788738153.006599 read 0x327000 thru 0x32f991 (0x8992 bytes) 000028[1] 1788738153.015177 mapwrite 0x0c0000 thru 0x0cdac0 (0xdac1 bytes) 000029[0] 1788738153.040381 mapwrite 0x758000 thru 0x75bb13 (0x3b14 bytes) 000030[0] 1788738153.067657 punch from 0x6b4e4a to 0x6c20bb (0xd272 bytes) 000032[1] 1788738153.072761 trunc from 0x969f82 to 0x59c849 (0x3cd738 bytes) 000033[0] 1788738153.083767 punch from 0x48443e to 0x491dbb (0xd97e bytes) 000034[1] 1788738153.087437 trunc from 0x59c84a to 0x0430ec (0x55975d bytes) 000041[1] 1788738153.107586 write 0x74b000 thru 0x74c342 (0x1343 bytes) 000043[0] 1788738153.114115 read 0x437000 thru 0x43fb32 (0x8b33 bytes) 000049[0] 1788738153.121965 mapread 0x3d4000 thru 0x3dca90 (0x8a91 bytes) 000050[0] 1788738153.127262 mapread 0x405000 thru 0x40cbba (0x7bbb bytes) 000051[1] 1788738153.129782 punch from 0x090f46 to 0x09137a (0x435 bytes) 000052[0] 1788738153.137270 mapwrite 0x9aa000 thru 0x9b9c7a (0xfc7b bytes) 000054[0] 1788738153.168326 trunc from 0x9b9c7b to 0x25942d (0x76084d bytes) 000055[0] 1788738153.175937 mapread 0x010000 thru 0x01011d (0x11e bytes) 000058[1] 1788738153.179285 punch from 0x2412b5 to 0x241a71 (0x7bd bytes) 000061[1] 1788738153.194794 trunc from 0x25942e to 0x5d0c81 (0x377854 bytes) 000063[1] 1788738153.205378 read 0x142000 thru 0x146859 (0x485a bytes) 000064[0] 1788738153.207733 mapwrite 0x778000 thru 0x783d61 (0xbd62 bytes) truncating to largest ever: 0x974504 000068[1] 1788738153.244091 trunc from 0x783d62 to 0x974503 (0x1f07a2 bytes) 000070[1] 1788738153.255785 write 0x1e2000 thru 0x1eb054 (0x9055 bytes) 000072[1] 1788738153.261008 write 0x2ef000 thru 0x2f9a71 (0xaa72 bytes) 000073[0] 1788738153.265859 write 0x596000 thru 0x5a17a2 (0xb7a3 bytes) 000075[1] 1788738153.279805 punch from 0x4f434a to 0x501779 (0xd430 bytes) 000076[1] 1788738153.288091 mapread 0x18c000 thru 0x1911f9 (0x51fa bytes) 000077[0] 1788738153.294343 trunc from 0x974504 to 0x30f8b8 (0x664c4b bytes) 000079[0] 1788738153.314003 trunc from 0x30f8b9 to 0x6daf9c (0x3cb6e4 bytes) 000080[0] 1788738153.324037 write 0x3dd000 thru 0x3e39ab (0x69ac bytes) 000081[0] 1788738153.329174 mapread 0x160000 thru 0x16784c (0x784d bytes) 000082[0] 1788738153.334066 mapwrite 0x579000 thru 0x586d81 (0xdd82 bytes) 000085[1] 1788738153.365448 mapread 0x158000 thru 0x15bdc8 (0x3dc9 bytes) 000086[1] 1788738153.372200 mapwrite 0x40e000 thru 0x4172de (0x92df bytes) 000087[0] 1788738153.396811 mapread 0x5fe000 thru 0x6073fa (0x93fb bytes) 000088[1] 1788738153.407008 read 0x4c2000 thru 0x4cf9ae (0xd9af bytes) 000089[1] 1788738153.411284 read 0x234000 thru 0x23e904 (0xa905 bytes) 000094[0] 1788738153.414574 trunc from 0x6daf9d to 0x2a632b (0x434c71 bytes) 000096[1] 1788738153.427081 punch from 0x1cfc30 to 0x1d95e6 (0x99b7 bytes) 000098[1] 1788738153.437509 mapwrite 0x520000 thru 0x52c087 (0xc088 bytes) 000103[1] 1788738153.475985 mapread 0x25b000 thru 0x269a4b (0xea4c bytes) 000105[0] 1788738153.486553 read 0x4d6000 thru 0x4df715 (0x9716 bytes) 000106[1] 1788738153.492151 mapwrite 0x761000 thru 0x764200 (0x3201 bytes) 000107[1] 1788738153.515302 punch from 0x48fa22 to 0x492a41 (0x3020 bytes) 000108[1] 1788738153.516861 write 0x584000 thru 0x58d11a (0x911b bytes) 000111[0] 1788738153.528356 mapwrite 0x2fd000 thru 0x2fe7b1 (0x17b2 bytes) 000113[0] 1788738153.552504 punch from 0x6e971f to 0x6f19d4 (0x82b6 bytes) 000114[1] 1788738153.557413 read 0x705000 thru 0x70c187 (0x7188 bytes) 000115[1] 1788738153.562093 punch from 0x5e20b9 to 0x5eded8 (0xbe20 bytes) 000118[0] 1788738153.569364 mapwrite 0x316000 thru 0x319c65 (0x3c66 bytes) 000119[1] 1788738153.594095 read 0x5a9000 thru 0x5b441c (0xb41d bytes) 000120[0] 1788738153.599380 mapread 0x59f000 thru 0x5a8afc (0x9afd bytes) 000121[0] 1788738153.612042 mapread 0x603000 thru 0x60b03a (0x803b bytes) 000122[0] 1788738153.615203 write 0x640000 thru 0x64a901 (0xa902 bytes) 000124[1] 1788738153.620210 mapwrite 0x7d3000 thru 0x7d6e44 (0x3e45 bytes) 000125[0] 1788738153.650366 mapread 0x42f000 thru 0x43d5ec (0xe5ed bytes) 000128[0] 1788738153.664674 mapread 0x296000 thru 0x2a4054 (0xe055 bytes) 000129[0] 1788738153.672102 mapwrite 0x447000 thru 0x44fa3f (0x8a40 bytes) 000131[0] 1788738153.713351 write 0x6e8000 thru 0x6f308a (0xb08b bytes) 000132[0] 1788738153.715902 mapread 0x4bb000 thru 0x4bebe1 (0x3be2 bytes) 000134[0] 1788738153.722939 mapwrite 0x5ea000 thru 0x5ed7f6 (0x37f7 bytes) 000136[1] 1788738153.747741 read 0x6bc000 thru 0x6cbaa5 (0xfaa6 bytes) 000137[1] 1788738153.754203 write 0x429000 thru 0x430982 (0x7983 bytes) 000138[1] 1788738153.758678 trunc from 0x7d6e45 to 0x93fdc4 (0x168f80 bytes) 000139[0] 1788738153.765950 mapwrite 0x730000 thru 0x730376 (0x377 bytes) 000142[1] 1788738153.797907 trunc from 0x93fdc5 to 0x3af6a1 (0x590723 bytes) 000146[1] 1788738153.813312 trunc from 0x3af6a2 to 0x691f76 (0x2e28d5 bytes) 000147[0] 1788738153.819696 trunc from 0x691f77 to 0x928ff1 (0x29707b bytes) 000148[1] 1788738153.827708 write 0x1bd000 thru 0x1cb542 (0xe543 bytes) 000149[0] 1788738153.833447 trunc from 0x928ff2 to 0x644637 (0x2e49ba bytes) 000150[0] 1788738153.846811 mapread 0x015000 thru 0x01c792 (0x7793 bytes) 000151[0] 1788738153.856279 read 0x4b3000 thru 0x4c19aa (0xe9ab bytes) 000153[1] 1788738153.860994 punch from 0x5929a7 to 0x59eb34 (0xc18e bytes) 000154[1] 1788738153.865537 mapread 0x1a7000 thru 0x1b1aaa (0xaaab bytes) 000156[0] 1788738153.874730 read 0x008000 thru 0x0081fd (0x1fe bytes) 000161[1] 1788738153.881015 mapread 0x51c000 thru 0x52bc4e (0xfc4f bytes) 000163[0] 1788738153.888646 punch from 0x0bced5 to 0x0be626 (0x1752 bytes) 000164[0] 1788738153.896422 write 0x800000 thru 0x804e41 (0x4e42 bytes) 000165[1] 1788738153.901370 mapread 0x4eb000 thru 0x4f002a (0x502b bytes) 000166[1] 1788738153.910143 mapread 0x0fc000 thru 0x1031b2 (0x71b3 bytes) 000167[0] 1788738153.915836 read 0x1c7000 thru 0x1d1310 (0xa311 bytes) 000168[0] 1788738153.919067 punch from 0x0f9c7d to 0x104124 (0xa4a8 bytes) 000170[1] 1788738153.929949 mapread 0x333000 thru 0x341624 (0xe625 bytes) 000175[1] 1788738153.943177 punch from 0x5dc72e to 0x5e8661 (0xbf34 bytes) 000180[0] 1788738153.948062 trunc from 0x804e42 to 0x86a55c (0x6571b bytes) 000182[0] 1788738153.959056 trunc from 0x86a55d to 0x826c5a (0x43902 bytes) 000185[0] 1788738153.968072 mapread 0x3fb000 thru 0x401529 (0x652a bytes) 000188[1] 1788738153.977376 write 0x69d000 thru 0x69eb61 (0x1b62 bytes) 000190[0] 1788738153.985650 punch from 0x43c220 to 0x44148d (0x526e bytes) 000192[0] 1788738153.994303 write 0x253000 thru 0x260410 (0xd411 bytes) 000193[1] 1788738153.999786 punch from 0x39e483 to 0x3aab3e (0xc6bc bytes) 000196[0] 1788738154.012961 write 0x68b000 thru 0x68e783 (0x3784 bytes) 000197[1] 1788738154.017598 read 0x4e0000 thru 0x4e5a3d (0x5a3e bytes) 000198[1] 1788738154.023862 trunc from 0x826c5b to 0x521226 (0x305a34 bytes) 000199[1] 1788738154.031151 punch from 0x182625 to 0x18c506 (0x9ee2 bytes) 000203[0] 1788738154.037076 write 0x6aa000 thru 0x6b99e7 (0xf9e8 bytes) 000206[0] 1788738154.045142 punch from 0x5cfbc1 to 0x5d1e39 (0x2279 bytes) 000207[0] 1788738154.046417 punch from 0x285be8 to 0x294b3f (0xef58 bytes) 000209[0] 1788738154.051592 read 0x0d8000 thru 0x0e13f4 (0x93f5 bytes) 000211[1] 1788738154.060584 read 0x361000 thru 0x363709 (0x270a bytes) 000213[1] 1788738154.068002 read 0x52c000 thru 0x538639 (0xc63a bytes) 000214[0] 1788738154.070657 trunc from 0x6b99e8 to 0x882be8 (0x1c9201 bytes) 000215[1] 1788738154.078525 punch from 0x712132 to 0x7126ac (0x57b bytes) 000216[1] 1788738154.085854 mapwrite 0x70b000 thru 0x70cf94 (0x1f95 bytes) 000217[1] 1788738154.103316 mapwrite 0x087000 thru 0x08b9cb (0x49cc bytes) 000219[0] 1788738154.125973 punch from 0x3ecc41 to 0x3fbabb (0xee7b bytes) 000221[0] 1788738154.130143 mapread 0x395000 thru 0x3a27a7 (0xd7a8 bytes) 000222[1] 1788738154.140684 punch from 0x731fed to 0x733038 (0x104c bytes) 000223[1] 1788738154.144431 mapread 0x671000 thru 0x678632 (0x7633 bytes) 000225[0] 1788738154.154973 mapwrite 0x357000 thru 0x365a0b (0xea0c bytes) 000226[0] 1788738154.182511 write 0x379000 thru 0x388c6c (0xfc6d bytes) 000227[0] 1788738154.186159 trunc from 0x882be9 to 0x152b51 (0x730097 bytes) 000228[1] 1788738154.191724 mapwrite 0x702000 thru 0x711b14 (0xfb15 bytes) 000231[1] 1788738154.228213 mapwrite 0x19b000 thru 0x1a91f1 (0xe1f2 bytes) 000232[0] 1788738154.250038 mapread 0x2cf000 thru 0x2de193 (0xf194 bytes) 000234[1] 1788738154.261640 trunc from 0x711b15 to 0x54ab92 (0x1c6f82 bytes) 000236[1] 1788738154.275518 write 0x25d000 thru 0x26a53f (0xd540 bytes) 000238[0] 1788738154.281508 punch from 0x06014d to 0x0665f0 (0x64a4 bytes) 000240[0] 1788738154.294893 mapwrite 0x15d000 thru 0x1686a1 (0xb6a2 bytes) 000241[1] 1788738154.324085 mapwrite 0x5e8000 thru 0x5ea81a (0x281b bytes) 000242[0] 1788738154.349112 punch from 0x4f3085 to 0x50271e (0xf69a bytes) 000243[0] 1788738154.356703 punch from 0x301aa3 to 0x30f3ef (0xd94d bytes) 000244[0] 1788738154.359207 read 0x170000 thru 0x171b87 (0x1b88 bytes) 000249[0] 1788738154.375635 read 0x445000 thru 0x4545bc (0xf5bd bytes) 000251[1] 1788738154.378706 write 0x12a000 thru 0x133a74 (0x9a75 bytes) 000252[0] 1788738154.383915 read 0x525000 thru 0x529e5b (0x4e5c bytes) 000254[0] 1788738154.391574 mapwrite 0x426000 thru 0x42ab53 (0x4b54 bytes) 000257[1] 1788738154.424451 mapwrite 0x2c0000 thru 0x2c0d50 (0xd51 bytes) 000258[0] 1788738154.441716 mapread 0x5be000 thru 0x5c805d (0xa05e bytes) 000259[0] 1788738154.452029 mapwrite 0x599000 thru 0x59ebd8 (0x5bd9 bytes) 000262[1] 1788738154.474174 write 0x3a8000 thru 0x3b28e1 (0xa8e2 bytes) 000263[1] 1788738154.478784 punch from 0x2bcb5e to 0x2c6a58 (0x9efb bytes) 000264[0] 1788738154.480296 write 0x073000 thru 0x0809df (0xd9e0 bytes) 000267[1] 1788738154.494354 write 0x365000 thru 0x372cea (0xdceb bytes) 000268[1] 1788738154.500932 write 0x285000 thru 0x28ce11 (0x7e12 bytes) 000270[1] 1788738154.506608 punch from 0x30309a to 0x3111c1 (0xe128 bytes) 000271[0] 1788738154.510030 write 0x7c1000 thru 0x7c3e73 (0x2e74 bytes) 000272[0] 1788738154.513761 punch from 0x56588a to 0x570880 (0xaff7 bytes) 000273[0] 1788738154.515206 mapread 0x67b000 thru 0x67fdc0 (0x4dc1 bytes) 000274[1] 1788738154.521507 read 0x69b000 thru 0x69f2c1 (0x42c2 bytes) 000275[0] 1788738154.528411 mapread 0x0bd000 thru 0x0bd3c0 (0x3c1 bytes) 000276[1] 1788738154.529946 mapread 0x4b1000 thru 0x4b7de0 (0x6de1 bytes) 000277[1] 1788738154.535393 read 0x1d8000 thru 0x1da372 (0x2373 bytes) 000278[1] 1788738154.537254 punch from 0x741284 to 0x748471 (0x71ee bytes) 000281[1] 1788738154.549773 write 0x9e7000 thru 0x9f3887 (0xc888 bytes) 000282[0] 1788738154.555111 write 0x0e2000 thru 0x0e3122 (0x1123 bytes) 000286[1] 1788738154.569793 mapwrite 0x788000 thru 0x78ffd6 (0x7fd7 bytes) 000287[0] 1788738154.599799 punch from 0x6d1d1e to 0x6d7fa0 (0x6283 bytes) 000288[0] 1788738154.607511 punch from 0x9e8fb3 to 0x9eca3f (0x3a8d bytes) 000290[1] 1788738154.612920 mapread 0x148000 thru 0x154799 (0xc79a bytes) 000291[0] 1788738154.624826 trunc from 0x9f3888 to 0x5385b9 (0x4bb2ce bytes) 000294[0] 1788738154.641379 trunc from 0x5385ba to 0x1d91d8 (0x35f3e1 bytes) 000295[0] 1788738154.646697 mapwrite 0x2d4000 thru 0x2d678a (0x278b bytes) 000296[1] 1788738154.668280 punch from 0x049f32 to 0x0558bb (0xb98a bytes) 000297[0] 1788738154.676047 write 0x97f000 thru 0x98d0c6 (0xe0c7 bytes) 000298[0] 1788738154.686253 mapread 0x11e000 thru 0x12c3e1 (0xe3e2 bytes) 000299[1] 1788738154.696682 read 0x02e000 thru 0x037ba2 (0x9ba3 bytes) 000300[1] 1788738154.703673 punch from 0x28695e to 0x2869d8 (0x7b bytes) 000301[0] 1788738154.707411 trunc from 0x98d0c7 to 0x41952a (0x573b9c bytes) 000302[1] 1788738154.714990 read 0x2e4000 thru 0x2eef1a (0xaf1b bytes) 000303[1] 1788738154.720987 punch from 0x15c113 to 0x161e9e (0x5d8c bytes) 000304[0] 1788738154.724936 write 0x3a6000 thru 0x3b08e3 (0xa8e4 bytes) 000305[0] 1788738154.733018 read 0x008000 thru 0x0150be (0xd0bf bytes) 000308[0] 1788738154.739917 mapwrite 0x3c2000 thru 0x3cd1bd (0xb1be bytes) 000309[1] 1788738154.762044 punch from 0x0dbc48 to 0x0e27c6 (0x6b7f bytes) 000310[0] 1788738154.770183 trunc from 0x41952b to 0x1f2903 (0x226c27 bytes) 000311[0] 1788738154.777202 write 0x072000 thru 0x07eb85 (0xcb86 bytes) 000314[1] 1788738154.788187 read 0x0c2000 thru 0x0c3f81 (0x1f82 bytes) 000315[0] 1788738154.805591 write 0x3c5000 thru 0x3ccf5a (0x7f5b bytes) 000321[1] 1788738154.827726 trunc from 0x3ccf5b to 0x32d361 (0x9fbf9 bytes) 000322[0] 1788738154.837365 write 0x3f4000 thru 0x402f1b (0xef1c bytes) 000324[1] 1788738154.847726 mapwrite 0x3d9000 thru 0x3e24da (0x94db bytes) 000326[0] 1788738154.882565 write 0x9be000 thru 0x9c1f1e (0x3f1f bytes) 000327[0] 1788738154.893357 write 0x780000 thru 0x78d616 (0xd617 bytes) 000328[0] 1788738154.897031 mapwrite 0x9d9000 thru 0x9da89e (0x189f bytes) 000329[0] 1788738154.917585 mapwrite 0x68e000 thru 0x69841e (0xa41f bytes) 000334[0] 1788738154.949173 write 0x6ad000 thru 0x6ad9b7 (0x9b8 bytes) 000335[0] 1788738154.954692 mapread 0x718000 thru 0x719481 (0x1482 bytes) 000336[0] 1788738154.957717 mapread 0x752000 thru 0x75bd8f (0x9d90 bytes) 000337[1] 1788738154.962604 mapwrite 0x399000 thru 0x3a4a50 (0xba51 bytes) 000338[0] 1788738154.996277 punch from 0x5aa5eb to 0x5abfa6 (0x19bc bytes) 000339[0] 1788738155.004041 trunc from 0x9da89f to 0x29062a (0x74a274 bytes) 000341[0] 1788738155.009682 mapread 0x005000 thru 0x0070cd (0x20ce bytes) 000342[0] 1788738155.019019 punch from 0x160c98 to 0x163e44 (0x31ad bytes) 000343[0] 1788738155.024995 punch from 0x2810e1 to 0x29062a (0xf54a bytes) 000348[0] 1788738155.041063 read 0x1ad000 thru 0x1b6bda (0x9bdb bytes) 000350[0] 1788738155.057999 mapread 0x1d0000 thru 0x1d487a (0x487b bytes) 000352[1] 1788738155.062354 punch from 0x152659 to 0x15baad (0x9455 bytes) 000353[1] 1788738155.069258 mapread 0x177000 thru 0x185c89 (0xec8a bytes) 000359[1] 1788738155.099934 mapread 0x013000 thru 0x01582f (0x2830 bytes) 000360[1] 1788738155.104002 trunc from 0x29062b to 0x77c865 (0x4ec23b bytes) 000361[0] 1788738155.114378 write 0x3f5000 thru 0x400ee5 (0xbee6 bytes) 000362[0] 1788738155.129861 write 0x437000 thru 0x444114 (0xd115 bytes) 000363[0] 1788738155.134598 write 0x0d0000 thru 0x0d7245 (0x7246 bytes) 000364[0] 1788738155.141672 punch from 0x1958b0 to 0x19fd26 (0xa477 bytes) 000365[1] 1788738155.143278 mapread 0x503000 thru 0x507640 (0x4641 bytes) 000366[1] 1788738155.175164 read 0x727000 thru 0x727317 (0x318 bytes) 000368[0] 1788738155.176816 punch from 0x751b0e to 0x75d72b (0xbc1e bytes) 000369[1] 1788738155.187426 mapwrite 0x192000 thru 0x1a0c5a (0xec5b bytes) 000372[1] 1788738155.225949 mapwrite 0x9c2000 thru 0x9c3e17 (0x1e18 bytes) 000373[1] 1788738155.245645 mapwrite 0x7f9000 thru 0x7fbf27 (0x2f28 bytes) 000374[0] 1788738155.268528 trunc from 0x9c3e18 to 0x3073af (0x6bca68 bytes) 000379[0] 1788738155.284802 write 0x801000 thru 0x80fc6b (0xec6c bytes) 000380[0] 1788738155.287820 read 0x0fd000 thru 0x0ff262 (0x2263 bytes) 000382[1] 1788738155.297315 write 0x14e000 thru 0x15bb20 (0xdb21 bytes) 000383[1] 1788738155.308292 read 0x759000 thru 0x75d3c9 (0x43ca bytes) 000385[0] 1788738155.314074 mapread 0x0d4000 thru 0x0d86f9 (0x46fa bytes) 000387[0] 1788738155.326310 mapwrite 0x1bb000 thru 0x1c8a5c (0xda5d bytes) 000388[0] 1788738155.353225 trunc from 0x80fc6c to 0x62d057 (0x1e2c14 bytes) 000389[0] 1788738155.358556 write 0x8a2000 thru 0x8a94bf (0x74c0 bytes) 000391[1] 1788738155.364839 mapread 0x48b000 thru 0x493aa9 (0x8aaa bytes) 000393[0] 1788738155.377393 read 0x7fe000 thru 0x80353d (0x553e bytes) 000394[0] 1788738155.379442 read 0x423000 thru 0x429ad5 (0x6ad6 bytes) 000397[1] 1788738155.381941 write 0x182000 thru 0x184789 (0x278a bytes) 000399[1] 1788738155.389827 punch from 0x6d903e to 0x6da438 (0x13fb bytes) 000401[0] 1788738155.394871 trunc from 0x8a94c0 to 0x59999e (0x30fb21 bytes) 000403[1] 1788738155.403266 punch from 0x2c945c to 0x2ca167 (0xd0c bytes) 000404[1] 1788738155.410358 write 0x87a000 thru 0x87cbf8 (0x2bf9 bytes) 000408[1] 1788738155.427167 read 0x008000 thru 0x012ec4 (0xaec5 bytes) 000410[1] 1788738155.434751 trunc from 0x87cbf9 to 0x412f99 (0x469c5f bytes) 000414[1] 1788738155.440658 trunc from 0x412f9a to 0x67d29b (0x26a302 bytes) 000416[1] 1788738155.449629 mapwrite 0x3b0000 thru 0x3b127a (0x127b bytes) 000421[1] 1788738155.472259 punch from 0x101463 to 0x10b2ee (0x9e8c bytes) 000422[1] 1788738155.473785 mapwrite 0x478000 thru 0x47ee60 (0x6e61 bytes) 000425[1] 1788738155.493645 read 0x09e000 thru 0x09f537 (0x1538 bytes) 000426[1] 1788738155.499092 trunc from 0x67d29c to 0x60d13b (0x70160 bytes) 000427[0] 1788738155.507391 write 0x153000 thru 0x161338 (0xe339 bytes) 000428[1] 1788738155.511641 trunc from 0x60d13c to 0x8c80cc (0x2baf91 bytes) 000432[0] 1788738155.530502 mapread 0x19e000 thru 0x1a7253 (0x9254 bytes) 000434[1] 1788738155.540347 write 0x673000 thru 0x6787a4 (0x57a5 bytes) 000435[1] 1788738155.547879 write 0x278000 thru 0x279e03 (0x1e04 bytes) 000436[0] 1788738155.549499 read 0x69e000 thru 0x6a334e (0x534f bytes) 000438[1] 1788738155.555934 write 0x3e1000 thru 0x3eb4cd (0xa4ce bytes) 000441[0] 1788738155.566878 punch from 0x4ae570 to 0x4b3fa4 (0x5a35 bytes) 000442[1] 1788738155.572577 write 0x0aa000 thru 0x0b8050 (0xe051 bytes) 000443[0] 1788738155.580098 read 0x7dd000 thru 0x7ebcaa (0xecab bytes) 000444[0] 1788738155.587440 mapwrite 0x1e1000 thru 0x1ef9b3 (0xe9b4 bytes) 000447[0] 1788738155.620240 write 0x1d4000 thru 0x1df29a (0xb29b bytes) 000448[1] 1788738155.624766 write 0x26a000 thru 0x26f5c1 (0x55c2 bytes) 000449[0] 1788738155.631258 mapread 0x88c000 thru 0x8908c3 (0x48c4 bytes) 000450[0] 1788738155.641869 trunc from 0x8c80cd to 0x1afaf3 (0x7185d9 bytes) 000452[1] 1788738155.658138 read 0x07f000 thru 0x08b45b (0xc45c bytes) 000454[1] 1788738155.672764 read 0x040000 thru 0x04669e (0x669f bytes) 000456[1] 1788738155.673343 read 0x0a1000 thru 0x0a1775 (0x776 bytes) 000459[0] 1788738155.674114 mapread 0x036000 thru 0x043985 (0xd986 bytes) 000461[1] 1788738155.681924 mapread 0x081000 thru 0x081002 (0x3 bytes) 000462[1] 1788738155.682343 write 0x8f9000 thru 0x8fe192 (0x5193 bytes) 000463[1] 1788738155.693923 punch from 0x037b04 to 0x047038 (0xf535 bytes) truncating to largest ever: 0x9e1bf5 000466[0] 1788738155.703127 trunc from 0x8fe193 to 0x9e1bf4 (0xe3a62 bytes) 000467[1] 1788738155.712146 punch from 0x50db11 to 0x51acbf (0xd1af bytes) 000468[0] 1788738155.716514 trunc from 0x9e1bf5 to 0x9591d0 (0x88a24 bytes) 000470[0] 1788738155.730375 read_OD 0x6ca000 thru 0x6d1fff (0x8000 bytes) 000473[1] 1788738155.735398 mapwrite 0x054000 thru 0x05a3c2 (0x63c3 bytes) 000474[0] 1788738155.771630 punch from 0x81e679 to 0x8233aa (0x4d32 bytes) 000475[0] 1788738155.783661 punch from 0x89f9f7 to 0x8adbfe (0xe208 bytes) 000476[1] 1788738155.788917 mapwrite 0x6fd000 thru 0x7037f3 (0x67f4 bytes) 000477[0] 1788738155.833007 write_OD 0x1cf000 thru 0x1ddfff (0xf000 bytes) 000479[1] 1788738155.851740 punch from 0x1d7693 to 0x1dfe8e (0x87fc bytes) 000480[0] 1788738155.859267 punch from 0x3455ee to 0x34deb9 (0x88cc bytes) 000482[0] 1788738155.866039 write_OD 0x72d000 thru 0x732fff (0x6000 bytes) 000484[1] 1788738155.882657 punch from 0x673ec1 to 0x67c7e1 (0x8921 bytes) 000485[1] 1788738155.890591 mapwrite 0x536000 thru 0x53d12c (0x712d bytes) 000487[0] 1788738155.919041 trunc from 0x9591d1 to 0x70a89a (0x24e936 bytes) 000489[1] 1788738155.926799 punch from 0x17c13c to 0x18b1c9 (0xf08e bytes) 000491[0] 1788738155.935707 punch from 0x44d352 to 0x44f399 (0x2048 bytes) 000492[1] 1788738155.939243 mapwrite 0x92e000 thru 0x931a34 (0x3a35 bytes) 000498[0] 1788738155.975845 punch from 0x7ad88c to 0x7ba8a1 (0xd016 bytes) 000499[1] 1788738155.984499 read 0x928000 thru 0x92cb2e (0x4b2f bytes) All operations completed A-OK! PASS 16a (11s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 19:42:37 (1788738157) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0955736 s, 87.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0716819 s, 117 MB/s Chance of close/open is 1 in 50 Seed set to 8158 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x97b76c 000001[1] 1788738158.872781 trunc from 00000000 to 0x97b76b (0x97b76c bytes) 000002[0] 1788738158.879989 trunc from 0x97b76c to 0x4d7a0e (0x4a3d5d bytes) 000005[1] 1788738158.900662 mapread 0x0d7e64 thru 0x0d8f39 (0x10d6 bytes) 000006[0] 1788738158.912395 trunc from 0x4d7a0f to 0x157252 (0x3807bc bytes) 000007[0] 1788738158.920580 read 0x0b45b5 thru 0x0b489c (0x2e8 bytes) 000008[0] 1788738158.927411 punch from 0x07c3d2 to 0x07db69 (0x1798 bytes) 000010[1] 1788738158.936650 read 0x020887 thru 0x020eae (0x628 bytes) 000014[1] 1788738158.954174 mapread 0x0b6e9d thru 0x0b8010 (0x1174 bytes) 000016[1] 1788738158.966441 punch from 0x145b93 to 0x14711d (0x158b bytes) 000017[0] 1788738158.974215 read 0x0bfc52 thru 0x0c11c5 (0x1574 bytes) 000019[1] 1788738158.994547 read 0x04a09d thru 0x04a1c0 (0x124 bytes) 000020[0] 1788738159.007145 read 0x106cce thru 0x108a14 (0x1d47 bytes) 000021[0] 1788738159.011986 mapwrite 0x625a18 thru 0x6263b3 (0x99c bytes) 000025[1] 1788738159.094762 punch from 0x40aef1 to 0x40c01c (0x112c bytes) 000026[0] 1788738159.105605 trunc from 0x6263b4 to 0x7eef3f (0x1c8b8c bytes) 000028[1] 1788738159.114737 mapwrite 0x6327f4 thru 0x6341cd (0x19da bytes) 000029[0] 1788738159.140595 trunc from 0x7eef40 to 0x24e8ab (0x5a0694 bytes) 000035[1] 1788738159.165093 mapwrite 0x96fd3a thru 0x970b56 (0xe1d bytes) 000036[1] 1788738159.248044 read 0x5a03e9 thru 0x5a21a5 (0x1dbd bytes) 000040[1] 1788738159.262694 punch from 0x5231b5 to 0x52492d (0x1779 bytes) 000041[1] 1788738159.268085 punch from 0x34b0a7 to 0x34c96d (0x18c7 bytes) 000042[0] 1788738159.269637 mapwrite 0x764c1f thru 0x765c5a (0x103c bytes) truncating to largest ever: 0x99ed7f 000043[1] 1788738159.308751 trunc from 0x970b57 to 0x99ed7e (0x2e228 bytes) 000044[1] 1788738159.323636 punch from 0x2d4443 to 0x2d6076 (0x1c34 bytes) 000046[1] 1788738159.332217 mapread 0x9782d5 thru 0x9793ad (0x10d9 bytes) 000047[1] 1788738159.340674 mapwrite 0x7dad57 thru 0x7dbed6 (0x1180 bytes) 000049[0] 1788738159.362188 punch from 0x4dfaa8 to 0x4e0d9a (0x12f3 bytes) 000050[1] 1788738159.370428 mapread 0x08aa48 thru 0x08c74e (0x1d07 bytes) 000051[0] 1788738159.383071 write 0x1dc32c thru 0x1dd443 (0x1118 bytes) 000053[1] 1788738159.398826 punch from 0x2f9438 to 0x2fac99 (0x1862 bytes) 000054[1] 1788738159.408654 trunc from 0x99ed7f to 0x4be788 (0x4e05f6 bytes) 000055[1] 1788738159.421842 mapwrite 0x22c565 thru 0x22d5bf (0x105b bytes) 000057[0] 1788738159.441116 write 0x89720b thru 0x898751 (0x1547 bytes) 000060[0] 1788738159.459074 write 0x131a4d thru 0x132c5f (0x1213 bytes) 000061[1] 1788738159.466013 mapwrite 0x4c50e9 thru 0x4c6fd9 (0x1ef1 bytes) 000063[1] 1788738159.498668 read 0x3945c7 thru 0x39467b (0xb5 bytes) 000064[0] 1788738159.506410 punch from 0x6e5aba to 0x6e6286 (0x7cd bytes) 000067[1] 1788738159.521178 mapread 0x1c7e28 thru 0x1c8ae7 (0xcc0 bytes) 000068[0] 1788738159.531044 trunc from 0x898752 to 0x079f55 (0x81e7fc bytes) 000069[0] 1788738159.545658 punch from 0x036e7c to 0x038642 (0x17c7 bytes) 000070[0] 1788738159.549165 mapread 0x0058bd thru 0x006fb7 (0x16fb bytes) 000072[0] 1788738159.558134 write 0x9db143 thru 0x9dc077 (0xf35 bytes) 000073[1] 1788738159.561328 write 0x0f0a9a thru 0x0f177c (0xce3 bytes) 000075[1] 1788738159.576161 mapwrite 0x72b846 thru 0x72d293 (0x1a4e bytes) 000076[1] 1788738159.602687 trunc from 0x9dc078 to 0x8be41d (0x11dc5a bytes) 000078[0] 1788738159.616098 read 0x12ba95 thru 0x12cdc9 (0x1335 bytes) 000079[1] 1788738159.623338 punch from 0x5e33cb to 0x5e4f1a (0x1b50 bytes) 000080[0] 1788738159.624789 write 0x63818f thru 0x639420 (0x1292 bytes) 000082[1] 1788738159.630299 write 0x28cb0a thru 0x28d07e (0x575 bytes) 000083[0] 1788738159.638079 read 0x4775e2 thru 0x478b26 (0x1545 bytes) 000084[0] 1788738159.643391 write 0x5cb69e thru 0x5cb838 (0x19b bytes) 000085[0] 1788738159.648588 mapread 0x4cf5ef thru 0x4cffac (0x9be bytes) 000086[1] 1788738159.655499 mapwrite 0x35ac08 thru 0x35c03a (0x1433 bytes) 000087[0] 1788738159.683084 write 0x600499 thru 0x60233f (0x1ea7 bytes) 000088[1] 1788738159.690657 trunc from 0x8be41e to 0x4a576b (0x418cb2 bytes) 000089[0] 1788738159.705676 mapread 0x3cc022 thru 0x3cc27c (0x25b bytes) 000090[1] 1788738159.715560 read 0x0a8e9d thru 0x0a9d91 (0xef5 bytes) 000091[0] 1788738159.717658 mapwrite 0x791fff thru 0x793211 (0x1213 bytes) 000092[1] 1788738159.751096 mapwrite 0x2df3ba thru 0x2df9e1 (0x628 bytes) 000094[1] 1788738159.778921 mapread 0x4820e1 thru 0x483a24 (0x1944 bytes) 000095[1] 1788738159.786101 mapwrite 0x9ae560 thru 0x9aec3a (0x6db bytes) 000096[0] 1788738159.807502 mapwrite 0x364227 thru 0x36446a (0x244 bytes) 000097[1] 1788738159.830482 trunc from 0x9aec3b to 0x7c019f (0x1eea9b bytes) 000099[1] 1788738159.847041 trunc from 0x7c01a0 to 0x170b33 (0x64f66c bytes) 000103[1] 1788738159.859325 read 0x00fcce thru 0x0115b5 (0x18e8 bytes) 000104[1] 1788738159.876792 mapwrite 0x1f0e26 thru 0x1f199e (0xb79 bytes) 000105[0] 1788738159.901730 trunc from 0x1f199f to 0x6aa659 (0x4b8cbb bytes) 000106[0] 1788738159.914940 mapwrite 0x1f1126 thru 0x1f16f9 (0x5d4 bytes) 000110[0] 1788738159.933627 mapread 0x6a08f6 thru 0x6a194f (0x105a bytes) 000111[0] 1788738159.936868 trunc from 0x6aa65a to 0x0aa16f (0x6004ea bytes) 000112[0] 1788738159.942157 punch from 0x018592 to 0x01a047 (0x1ab6 bytes) 000113[0] 1788738159.943538 write 0x722ee3 thru 0x722f63 (0x81 bytes) 000114[1] 1788738159.945543 mapwrite 0x2fca6c thru 0x2fe06f (0x1604 bytes) 000115[0] 1788738159.991340 punch from 0x315c84 to 0x317806 (0x1b83 bytes) 000117[1] 1788738159.996743 write 0x3f4440 thru 0x3f5af9 (0x16ba bytes) 000119[0] 1788738160.006784 read 0x4b5040 thru 0x4b6f60 (0x1f21 bytes) 000120[1] 1788738160.012148 mapread 0x0de856 thru 0x0dff15 (0x16c0 bytes) 000121[1] 1788738160.026809 punch from 0x21e9ba to 0x2204e0 (0x1b27 bytes) 000123[1] 1788738160.035699 read 0x194f5c thru 0x196d8b (0x1e30 bytes) 000124[0] 1788738160.041050 mapwrite 0x9dd7aa thru 0x9de49d (0xcf4 bytes) 000126[1] 1788738160.071071 mapread 0x6edb32 thru 0x6ef0cc (0x159b bytes) 000127[0] 1788738160.082927 trunc from 0x9de49e to 0x5b13b5 (0x42d0e8 bytes) 000128[0] 1788738160.091209 punch from 0x2777fa to 0x278c78 (0x147f bytes) 000131[1] 1788738160.101991 mapread 0x39f6b1 thru 0x3a0ce3 (0x1633 bytes) 000132[1] 1788738160.110441 punch from 0x4a2234 to 0x4a311c (0xee9 bytes) 000136[1] 1788738160.126079 trunc from 0x5b13b6 to 0x917019 (0x365c64 bytes) 000137[1] 1788738160.139408 trunc from 0x91701a to 0x1d3d5a (0x7432bf bytes) 000139[0] 1788738160.151152 punch from 0x191b58 to 0x191c2c (0xd5 bytes) 000140[0] 1788738160.158236 write 0x170c42 thru 0x1715c8 (0x987 bytes) 000141[1] 1788738160.164763 read 0x064319 thru 0x064dd0 (0xab8 bytes) 000142[0] 1788738160.187138 trunc from 0x1d3d5b to 0x1566bc (0x7d69e bytes) 000145[0] 1788738160.213266 mapread 0x0e1320 thru 0x0e148a (0x16b bytes) 000146[1] 1788738160.219753 mapread 0x139fa3 thru 0x13ac1c (0xc7a bytes) 000147[0] 1788738160.231743 mapread 0x0cbfbf thru 0x0cc038 (0x7a bytes) 000149[0] 1788738160.246993 trunc from 0x1566bd to 0x9578c5 (0x801209 bytes) 000150[0] 1788738160.259143 mapread 0x516a25 thru 0x516d6b (0x347 bytes) 000153[0] 1788738160.271230 punch from 0x170db9 to 0x171894 (0xadc bytes) 000154[1] 1788738160.272931 trunc from 0x9578c6 to 0x422a99 (0x534e2c bytes) 000155[0] 1788738160.282865 punch from 0x151669 to 0x1522c6 (0xc5e bytes) 000156[0] 1788738160.291512 write 0x0b2835 thru 0x0b31b4 (0x980 bytes) 000157[1] 1788738160.298152 mapread 0x313d89 thru 0x3140a4 (0x31c bytes) 000158[1] 1788738160.320664 write 0x483216 thru 0x484b27 (0x1912 bytes) 000159[0] 1788738160.326342 read 0x102ca9 thru 0x104bef (0x1f47 bytes) 000161[1] 1788738160.328720 read 0x3a0b4b thru 0x3a2a4c (0x1f02 bytes) 000162[1] 1788738160.329032 mapwrite 0x3188ba thru 0x319511 (0xc58 bytes) 000163[1] 1788738160.361759 mapwrite 0x263028 thru 0x264878 (0x1851 bytes) 000164[0] 1788738160.384785 trunc from 0x484b28 to 0x41f08c (0x65a9b bytes) 000165[0] 1788738160.396738 read 0x07d5f0 thru 0x07df43 (0x954 bytes) 000166[0] 1788738160.404213 write 0x190831 thru 0x1923c7 (0x1b97 bytes) 000167[0] 1788738160.411585 mapwrite 0x309dc4 thru 0x30ba01 (0x1c3e bytes) 000168[1] 1788738160.434434 trunc from 0x41f08d to 0x3adbdb (0x714b1 bytes) 000171[1] 1788738160.450836 write 0x6ce2e8 thru 0x6ce585 (0x29e bytes) 000175[0] 1788738160.453758 trunc from 0x6ce586 to 0x281b3e (0x44ca47 bytes) 000176[1] 1788738160.469938 write 0x0caf7f thru 0x0cb5d9 (0x65b bytes) 000177[1] 1788738160.479090 punch from 0x280416 to 0x2814b6 (0x10a1 bytes) 000179[0] 1788738160.487234 trunc from 0x281b3f to 0x8c1732 (0x63fbf4 bytes) 000181[0] 1788738160.502581 write 0x418a98 thru 0x418cd6 (0x23f bytes) 000184[0] 1788738160.513610 trunc from 0x8c1733 to 0x9512df (0x8fbad bytes) 000185[0] 1788738160.518854 write 0x16e08b thru 0x16e316 (0x28c bytes) 000186[1] 1788738160.520750 read 0x946257 thru 0x947154 (0xefe bytes) 000187[0] 1788738160.526576 punch from 0x06ba76 to 0x06c1f5 (0x780 bytes) 000189[1] 1788738160.535930 punch from 0x0775bf to 0x077960 (0x3a2 bytes) 000190[1] 1788738160.543259 read 0x388bd6 thru 0x3897c8 (0xbf3 bytes) 000193[0] 1788738160.556505 read 0x454d40 thru 0x455c55 (0xf16 bytes) 000194[1] 1788738160.562730 trunc from 0x9512e0 to 0x65f875 (0x2f1a6a bytes) 000195[1] 1788738160.572618 punch from 0x208a5f to 0x209342 (0x8e4 bytes) 000196[0] 1788738160.580379 write 0x4f18e2 thru 0x4f2a7b (0x119a bytes) 000201[0] 1788738160.605612 write 0x742c0c thru 0x744ba2 (0x1f97 bytes) 000202[1] 1788738160.609961 read 0x6a6794 thru 0x6a73e6 (0xc53 bytes) 000203[1] 1788738160.619436 mapwrite 0x44dca8 thru 0x44eb5c (0xeb5 bytes) 000204[1] 1788738160.637162 read 0x2bffa1 thru 0x2c1c6a (0x1cca bytes) 000205[1] 1788738160.644622 punch from 0x56a3e9 to 0x56a4c1 (0xd9 bytes) 000206[1] 1788738160.646424 mapread 0x62d63b thru 0x62df6a (0x930 bytes) 000208[1] 1788738160.657802 trunc from 0x744ba3 to 0x3ab58a (0x399618 bytes) 000210[0] 1788738160.669517 write 0x2144be thru 0x2146aa (0x1ed bytes) 000211[1] 1788738160.677550 punch from 0x0b03c6 to 0x0b1379 (0xfb4 bytes) 000212[0] 1788738160.686338 trunc from 0x3ab58b to 0x2d354e (0xd803c bytes) 000213[1] 1788738160.699491 read 0x1a5fda thru 0x1a7eea (0x1f11 bytes) 000216[1] 1788738160.727494 mapread 0x042deb thru 0x0435cd (0x7e3 bytes) 000218[1] 1788738160.737776 write 0x0c4160 thru 0x0c5b09 (0x19aa bytes) 000219[0] 1788738160.750051 mapwrite 0x6ff5b3 thru 0x700527 (0xf75 bytes) 000220[1] 1788738160.780905 write 0x96ecde thru 0x9704b8 (0x17db bytes) 000221[1] 1788738160.788486 trunc from 0x9704b9 to 0x4aea2b (0x4c1a8d bytes) 000222[0] 1788738160.794815 write 0x7ac9d6 thru 0x7ad771 (0xd9c bytes) 000225[0] 1788738160.810789 mapwrite 0x52a20d thru 0x52c157 (0x1f4b bytes) 000226[0] 1788738160.834619 trunc from 0x7ad772 to 0x891eef (0xe477e bytes) 000227[0] 1788738160.842900 trunc from 0x891ef0 to 0x773171 (0x11ed7e bytes) 000228[0] 1788738160.846918 mapwrite 0x06e2eb thru 0x06eec5 (0xbdb bytes) 000230[1] 1788738160.866322 punch from 0x24ba60 to 0x24d9c0 (0x1f61 bytes) 000231[1] 1788738160.873990 read 0x29e0e3 thru 0x29e5d0 (0x4ee bytes) 000232[1] 1788738160.879251 read 0x5255b4 thru 0x5262f2 (0xd3f bytes) 000236[0] 1788738160.895007 trunc from 0x773172 to 0x5aedbe (0x1c43b3 bytes) 000237[0] 1788738160.909142 mapwrite 0x825d3a thru 0x8279ae (0x1c75 bytes) 000238[0] 1788738160.938139 read 0x1d74c8 thru 0x1d78a5 (0x3de bytes) 000239[0] 1788738160.941674 mapwrite 0x273111 thru 0x273395 (0x285 bytes) 000240[1] 1788738160.954756 trunc from 0x8279af to 0x4e9f3a (0x33da74 bytes) 000241[1] 1788738160.965741 mapread 0x138f93 thru 0x1398bc (0x92a bytes) 000242[1] 1788738160.969135 trunc from 0x4e9f3b to 0x4edaa6 (0x3b6c bytes) 000243[1] 1788738160.977533 punch from 0x250e37 to 0x251c78 (0xe42 bytes) 000244[0] 1788738160.980439 mapwrite 0x53feba thru 0x540a97 (0xbde bytes) 000245[1] 1788738161.005852 read 0x05f76e thru 0x060197 (0xa2a bytes) 000247[1] 1788738161.011542 mapread 0x10ca70 thru 0x10deb0 (0x1441 bytes) 000249[1] 1788738161.024578 read 0x0af2b3 thru 0x0b016c (0xeba bytes) 000250[0] 1788738161.030192 mapwrite 0x1a0877 thru 0x1a1d1a (0x14a4 bytes) 000252[1] 1788738161.047655 write 0x090ac3 thru 0x090f63 (0x4a1 bytes) 000253[1] 1788738161.055813 mapread 0x14b2bf thru 0x14c44e (0x1190 bytes) 000254[1] 1788738161.063972 write 0x3a3dfb thru 0x3a3f45 (0x14b bytes) 000259[0] 1788738161.084123 read 0x0d6a5a thru 0x0d83d6 (0x197d bytes) 000261[1] 1788738161.090907 mapwrite 0x84d2fb thru 0x84e818 (0x151e bytes) 000262[1] 1788738161.121195 mapread 0x5d088f thru 0x5d16ed (0xe5f bytes) 000266[0] 1788738161.141169 punch from 0x605e2b to 0x60600b (0x1e1 bytes) 000268[0] 1788738161.153088 trunc from 0x84e819 to 0x72d6d7 (0x121141 bytes) 000269[1] 1788738161.164228 mapread 0x61150f thru 0x611ba2 (0x694 bytes) 000271[0] 1788738161.178662 write 0x5d5718 thru 0x5d691a (0x1203 bytes) 000272[0] 1788738161.185360 mapread 0x218b73 thru 0x219e3e (0x12cc bytes) 000274[0] 1788738161.192588 punch from 0x1e718c to 0x1e8db9 (0x1c2e bytes) 000275[0] 1788738161.198169 mapwrite 0x74518c thru 0x74599e (0x813 bytes) 000276[1] 1788738161.216203 mapwrite 0x5c0cde thru 0x5c2409 (0x172c bytes) 000277[0] 1788738161.242100 read 0x65756b thru 0x65845a (0xef0 bytes) 000278[1] 1788738161.250411 write 0x2d0c9c thru 0x2d1328 (0x68d bytes) 000279[1] 1788738161.255728 read 0x428ef5 thru 0x429b86 (0xc92 bytes) 000281[1] 1788738161.266356 punch from 0x3567d9 to 0x357619 (0xe41 bytes) 000282[1] 1788738161.271565 mapwrite 0x2140ff thru 0x215057 (0xf59 bytes) 000283[1] 1788738161.287055 trunc from 0x74599f to 0x07ca57 (0x6c8f47 bytes) 000284[0] 1788738161.299554 mapread 0x044325 thru 0x044507 (0x1e3 bytes) 000286[0] 1788738161.312887 mapwrite 0x09f78e thru 0x09f884 (0xf7 bytes) 000288[1] 1788738161.340758 punch from 0x03adb3 to 0x03cc28 (0x1e76 bytes) 000290[0] 1788738161.352919 punch from 0x071350 to 0x072dc0 (0x1a71 bytes) 000291[0] 1788738161.360805 trunc from 0x09f885 to 0x8f9221 (0x85999d bytes) 000296[0] 1788738161.379596 write 0x64ab0a thru 0x64b247 (0x73e bytes) 000299[0] 1788738161.392774 mapwrite 0x14adcd thru 0x14b49b (0x6cf bytes) 000301[1] 1788738161.413740 punch from 0x77634d to 0x776a04 (0x6b8 bytes) 000302[1] 1788738161.423538 trunc from 0x8f9222 to 0x21030e (0x6e8f13 bytes) 000303[1] 1788738161.431743 trunc from 0x21030f to 0x791a02 (0x5816f4 bytes) 000304[0] 1788738161.437107 read 0x0f0cfc thru 0x0f1dc3 (0x10c8 bytes) 000307[1] 1788738161.452153 read 0x398f6f thru 0x39a463 (0x14f5 bytes) 000309[1] 1788738161.461839 write 0x905823 thru 0x9073e6 (0x1bc4 bytes) 000310[1] 1788738161.470191 trunc from 0x9073e7 to 0x29f408 (0x667fde bytes) 000316[1] 1788738161.491436 write 0x362f58 thru 0x363435 (0x4de bytes) 000318[0] 1788738161.497114 read 0x18b44e thru 0x18bc74 (0x827 bytes) 000319[1] 1788738161.526912 mapwrite 0x7b34aa thru 0x7b437c (0xed3 bytes) 000321[1] 1788738161.554220 punch from 0x125add to 0x1264b1 (0x9d5 bytes) 000322[0] 1788738161.555761 punch from 0x747ade to 0x74876a (0xc8d bytes) 000323[0] 1788738161.563064 punch from 0x79c829 to 0x79d873 (0x104b bytes) 000326[0] 1788738161.575248 read 0x63b312 thru 0x63c89a (0x1589 bytes) 000328[1] 1788738161.585547 mapwrite 0x6eb571 thru 0x6ed260 (0x1cf0 bytes) 000329[0] 1788738161.604601 punch from 0x406152 to 0x40644f (0x2fe bytes) 000331[0] 1788738161.616947 mapread 0x7a4127 thru 0x7a51df (0x10b9 bytes) 000335[1] 1788738161.636888 write 0x64ecab thru 0x64fb08 (0xe5e bytes) 000338[0] 1788738161.650870 mapwrite 0x585936 thru 0x5867a9 (0xe74 bytes) 000339[1] 1788738161.674518 read 0x3b1c90 thru 0x3b39e4 (0x1d55 bytes) 000340[1] 1788738161.680240 write 0x8701ee thru 0x870e30 (0xc43 bytes) 000341[1] 1788738161.684600 read 0x7a8353 thru 0x7aa1b5 (0x1e63 bytes) 000345[0] 1788738161.694205 write 0x7452cd thru 0x7453ad (0xe1 bytes) 000347[1] 1788738161.705395 write 0x887b2c thru 0x888df9 (0x12ce bytes) 000349[0] 1788738161.713743 mapwrite 0x15bce4 thru 0x15d579 (0x1896 bytes) 000350[1] 1788738161.742299 punch from 0x5e70e8 to 0x5e8212 (0x112b bytes) 000351[0] 1788738161.751103 write 0x6c4dae thru 0x6c56ba (0x90d bytes) 000352[0] 1788738161.759507 punch from 0x237eec to 0x238003 (0x118 bytes) 000354[0] 1788738161.761232 punch from 0x838fe1 to 0x839a2f (0xa4f bytes) 000358[0] 1788738161.767898 mapwrite 0x30a142 thru 0x30b9dd (0x189c bytes) 000359[0] 1788738161.786877 mapread 0x28ca8a thru 0x28e27e (0x17f5 bytes) 000362[0] 1788738161.802416 mapwrite 0x3adcf5 thru 0x3ade78 (0x184 bytes) 000364[1] 1788738161.821122 mapread 0x7dd6a8 thru 0x7ddeef (0x848 bytes) 000366[0] 1788738161.831153 write 0x74912c thru 0x749ad9 (0x9ae bytes) 000367[0] 1788738161.839294 mapwrite 0x931674 thru 0x93183d (0x1ca bytes) 000368[1] 1788738161.858570 mapread 0x0d7407 thru 0x0d7f52 (0xb4c bytes) 000369[1] 1788738161.866252 mapwrite 0x381b66 thru 0x382005 (0x4a0 bytes) 000372[1] 1788738161.899060 trunc from 0x93183e to 0x38a967 (0x5a6ed6 bytes) 000374[1] 1788738161.909391 mapwrite 0x29a26f thru 0x29bfbc (0x1d4e bytes) 000375[0] 1788738161.929641 punch from 0x26b181 to 0x26cc1a (0x1a9a bytes) 000376[0] 1788738161.937549 mapread 0x1390f3 thru 0x13ab63 (0x1a71 bytes) 000377[1] 1788738161.946283 mapwrite 0x2fb6a1 thru 0x2fd0f3 (0x1a53 bytes) 000378[0] 1788738161.969671 mapread 0x280ea3 thru 0x282976 (0x1ad4 bytes) 000379[1] 1788738161.980965 mapread 0x0c0620 thru 0x0c2244 (0x1c25 bytes) 000382[0] 1788738161.997126 read 0x235b6c thru 0x23620f (0x6a4 bytes) 000383[1] 1788738162.017593 write 0x8d8b7e thru 0x8d8e10 (0x293 bytes) 000386[1] 1788738162.039867 trunc from 0x8d8e11 to 0x0ec616 (0x7ec7fa bytes) 000388[0] 1788738162.050373 read 0x06f767 thru 0x0706f1 (0xf8b bytes) 000389[0] 1788738162.062267 mapwrite 0x8046d9 thru 0x805f80 (0x18a8 bytes) 000393[0] 1788738162.141797 mapwrite 0x544b72 thru 0x545369 (0x7f8 bytes) 000396[0] 1788738162.165006 mapread 0x349c84 thru 0x34b7b5 (0x1b32 bytes) 000399[0] 1788738162.166498 mapread 0x7d1539 thru 0x7d1861 (0x329 bytes) 000400[0] 1788738162.167162 write 0x6b0f68 thru 0x6b1ebb (0xf54 bytes) 000402[1] 1788738162.176871 mapread 0x36b596 thru 0x36bdfc (0x867 bytes) 000406[0] 1788738162.200887 punch from 0x2836c8 to 0x2846af (0xfe8 bytes) 000407[0] 1788738162.205234 write 0x73f069 thru 0x73fb09 (0xaa1 bytes) 000409[0] 1788738162.206547 trunc from 0x805f81 to 0x6dd910 (0x128670 bytes) 000412[1] 1788738162.213624 trunc from 0x6dd911 to 0x96bc10 (0x28e300 bytes) 000413[0] 1788738162.231715 read 0x640be0 thru 0x641f24 (0x1345 bytes) truncating to largest ever: 0x9e29cf 000414[0] 1788738162.237443 trunc from 0x96bc11 to 0x9e29ce (0x76dbe bytes) 000415[1] 1788738162.246771 read 0x02a9da thru 0x02c60b (0x1c32 bytes) 000419[1] 1788738162.257100 mapread 0x105309 thru 0x107235 (0x1f2d bytes) 000420[0] 1788738162.263709 punch from 0x1dfb4c to 0x1e0a2d (0xee2 bytes) 000421[0] 1788738162.267729 trunc from 0x9e29cf to 0x652047 (0x390987 bytes) 000422[1] 1788738162.277039 mapwrite 0x550095 thru 0x55183d (0x17a9 bytes) 000424[0] 1788738162.306112 write 0x7addf1 thru 0x7ae0ac (0x2bc bytes) 000425[0] 1788738162.309627 punch from 0x296678 to 0x297b19 (0x14a2 bytes) 000428[1] 1788738162.316046 mapwrite 0x540ddb thru 0x54245b (0x1681 bytes) 000432[1] 1788738162.355099 write 0x192df1 thru 0x194b26 (0x1d36 bytes) 000434[0] 1788738162.368227 trunc from 0x7ae0ad to 0x969f30 (0x1bbe84 bytes) 000435[0] 1788738162.379223 read 0x7c04c2 thru 0x7c1fbc (0x1afb bytes) 000436[0] 1788738162.385749 trunc from 0x969f31 to 0x756dba (0x213176 bytes) 000440[1] 1788738162.391588 write 0x6b53f1 thru 0x6b6625 (0x1235 bytes) 000442[1] 1788738162.405995 mapread 0x2fe91a thru 0x2ff201 (0x8e8 bytes) 000443[0] 1788738162.416623 mapread 0x20b867 thru 0x20c5de (0xd78 bytes) 000445[1] 1788738162.425303 mapwrite 0x078119 thru 0x079054 (0xf3c bytes) 000446[1] 1788738162.443874 mapread 0x0fac07 thru 0x0facd5 (0xcf bytes) 000447[1] 1788738162.447185 mapwrite 0x319c1e thru 0x31ab18 (0xefb bytes) 000448[1] 1788738162.468360 punch from 0x740497 to 0x741355 (0xebf bytes) 000449[0] 1788738162.474741 trunc from 0x756dbb to 0x5047ef (0x2525cb bytes) 000451[0] 1788738162.487151 write 0x50e385 thru 0x50f06c (0xce8 bytes) 000453[1] 1788738162.497582 mapwrite 0x21e271 thru 0x21ec79 (0xa09 bytes) 000454[0] 1788738162.525137 read 0x2b1a0c thru 0x2b1d9b (0x390 bytes) 000456[0] 1788738162.534278 mapread 0x3d0a15 thru 0x3d0b6c (0x158 bytes) 000457[1] 1788738162.536214 mapwrite 0x04a022 thru 0x04a75e (0x73d bytes) 000458[0] 1788738162.557993 trunc from 0x50f06d to 0x74c7c4 (0x23d758 bytes) 000461[0] 1788738162.566442 read 0x189dcb thru 0x189ee7 (0x11d bytes) 000464[1] 1788738162.577365 read 0x383336 thru 0x384a39 (0x1704 bytes) 000465[0] 1788738162.582099 write 0x2a8c27 thru 0x2a98c7 (0xca1 bytes) 000466[1] 1788738162.590898 punch from 0x3cc7f9 to 0x3cd906 (0x110e bytes) 000468[0] 1788738162.604011 read 0x5dfcbc thru 0x5e04b1 (0x7f6 bytes) 000469[0] 1788738162.609068 read 0x7355db thru 0x736398 (0xdbe bytes) 000470[1] 1788738162.614159 mapwrite 0x5a2abf thru 0x5a4266 (0x17a8 bytes) 000471[1] 1788738162.635989 write 0x1a0b5c thru 0x1a0cab (0x150 bytes) 000476[1] 1788738162.651999 mapwrite 0x3d6628 thru 0x3d76ba (0x1093 bytes) 000477[0] 1788738162.666598 trunc from 0x74c7c5 to 0x847998 (0xfb1d4 bytes) 000478[0] 1788738162.676507 read 0x046eae thru 0x048833 (0x1986 bytes) 000480[0] 1788738162.687438 write 0x120d21 thru 0x122112 (0x13f2 bytes) 000482[0] 1788738162.693687 mapread 0x522c00 thru 0x523f7c (0x137d bytes) 000483[1] 1788738162.700127 trunc from 0x847999 to 0x27bbd0 (0x5cbdc8 bytes) 000484[1] 1788738162.711779 punch from 0x0bd640 to 0x0bdfec (0x9ad bytes) 000485[1] 1788738162.713334 punch from 0x17d13b to 0x17e85e (0x1724 bytes) 000488[0] 1788738162.721359 mapread 0x1289a0 thru 0x12998a (0xfeb bytes) 000490[1] 1788738162.736343 trunc from 0x27bbd1 to 0x3b66d2 (0x13ab02 bytes) 000493[1] 1788738162.749906 punch from 0x022a0f to 0x02445b (0x1a4d bytes) 000494[1] 1788738162.758231 write 0x32137d thru 0x321c31 (0x8b5 bytes) 000495[0] 1788738162.767088 mapwrite 0x3fe625 thru 0x3fe689 (0x65 bytes) 000497[0] 1788738162.795111 write 0x307200 thru 0x308997 (0x1798 bytes) 000498[1] 1788738162.799026 write 0x4b7472 thru 0x4b7ff0 (0xb7f bytes) 000499[1] 1788738162.806690 mapread 0x29d88d thru 0x29dcc3 (0x437 bytes) 000500[1] 1788738162.816045 punch from 0x060f04 to 0x062271 (0x136e bytes) All operations completed A-OK! PASS 16b (7s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 19:42:44 (1788738164) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (1s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 19:42:45 (1788738165) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.120484 s, 69.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0808412 s, 104 MB/s ldlm.namespaces.MGC192.168.202.131@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9b9e10447000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9b9e20121800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9b9e20121800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9b9e20121800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.64034 s, 39.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.711208 s, 147 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.853158 s, 123 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.47063 s, 71.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.60011 s, 65.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.10347 s, 95.0 MB/s PASS 16d (12s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 19:42:57 (1788738177) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0963198 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360214 s, 29.1 MB/s PASS 16e (3s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 19:43:00 (1788738180) TIMEOUT PASS 16f (22s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 19:43:22 (1788738202) TIMEOUT PASS 16g (22s) == sanityn test 16h: mmap read after truncate file ======= 19:43:44 (1788738224) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (3s) == sanityn test 16i: read after truncate file ============ 19:43:47 (1788738227) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (2s) == sanityn test 16j: race dio with buffered i/o ========== 19:43:49 (1788738229) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.402468 s, 65.1 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.411512 s, 63.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.55257 s, 47.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.395904 s, 66.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.512791 s, 51.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.491957 s, 53.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.595967 s, 44.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.460462 s, 56.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.548145 s, 47.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.635824 s, 41.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.678679 s, 38.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.523349 s, 50.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.56862 s, 46.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.693433 s, 37.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.731779 s, 35.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.690041 s, 38.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.705714 s, 37.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.823085 s, 31.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.844255 s, 31.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (9s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 19:43:58 (1788738238) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x1a7062c skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Sun Sep 6 19:44:08 EDT 2026 + fsx 2 done: rc=0 Sun Sep 6 19:44:08 EDT 2026 All operations completed A-OK! + fsx 3 done: rc=0 Sun Sep 6 19:44:08 EDT 2026 + fsx 4 done: rc=0 Sun Sep 6 19:44:08 EDT 2026 All operations completed A-OK! + fsx 5 done: rc=0 Sun Sep 6 19:44:08 EDT 2026 PASS 16k (13s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 19:44:11 (1788738251) fail_loc=0x8000030a PASS 17 (5s) == sanityn test 18: mmap sanity check =========================================================================================== 19:44:16 (1788738256) mmap test1: basic mmap operation (PASS, 0.024202s) mmap test2: MAP_PRIVATE not write back (PASS, 0.017329s) mmap test3: concurrent mmap ops on two nodes (PASS, 2.2148s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.1006s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.032099s) mmap test6: check mmap write/read content on two nodes (PASS, 0.040458s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.43711s) mmap test11: multi-thread shared mmap (PASS, 0.41507s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0167s) PASS 18 (12s) == sanityn test 19: test concurrent uncached read races ========================================================================= 19:44:28 (1788738268) oleg231-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (1s) == sanityn test 20: test extra readahead page left in cache ============================================================== 19:44:29 (1788738269) PASS 20 (2s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 19:44:31 (1788738271) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (3s) == sanityn test 23: others should see updated atime while another read============================================================== 19:44:34 (1788738274) now is 1788738274 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 new atime is 1788738335 PASS 23 (63s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 19:45:37 (1788738337) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 387 524.5K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 401 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 387 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 387 524.5K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 401 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 387 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.0M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537448 387 537061 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118001 401 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234883 387 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 387 524.5K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 401 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 387 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537448 387 537061 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118001 401 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234883 387 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537448 387 537061 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118001 401 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234883 387 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (2s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 19:45:40 (1788738340) PASS 24b (3s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 19:45:42 (1788738342) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (2s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 19:45:44 (1788738344) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (1s) == sanityn test 26a: allow mtime to get older ============ 19:45:45 (1788738345) utime: good mknod times 1788738344 <= 1788738345 <= 1788738345 for /mnt/lustre/f26a utime: good mknod times 1788738344 <= 1788738345 <= 1788738345 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (3s) == sanityn test 26b: sync mtime between ost and mds ====== 19:45:48 (1788738348) PASS 26b (4s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 19:45:52 (1788738352) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-06 19:45:53.000000000 -0400 Birth: 2026-09-06 19:45:52.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-06 19:45:53.000000000 -0400 Birth: 2026-09-06 19:45:52.000000000 -0400 PASS 26c (3s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 19:45:56 (1788738356) 0 PASS 30 (4s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 19:46:00 (1788738360) fail_loc=0x314 PASS 31a (3s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 19:46:03 (1788738363) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg231-server mds-ost sync done. sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.131@tcp:/lustre 7542784 39936 7496704 1% /mnt/lustre2 PASS 31b (18s) == sanityn test 31r: open-rename(replace) race =========== 19:46:21 (1788738381) -rw-r--r-- 1 root root 0 Sep 6 19:46 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 mv: cannot stat '/mnt/lustre2/f31r.sanityn.source': Input/output error PASS 31r (5s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 19:46:26 (1788738386) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43153 Terminated tail -f $tfile PASS 31s (3s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 19:46:29 (1788738389) PASS 31t (3s) == sanityn test 32b: lockless i/o ======================== 19:46:32 (1788738392) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (1s) == sanityn test 32c: contention detection ================ 19:46:33 (1788738393) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.472073 s, 111 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.400681 s, 131 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.898507 s, 58.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.979429 s, 53.5 MB/s contention_seconds 4, contention_events 20 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.936945 s, 56.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.0008 s, 52.4 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.376784 s, 139 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.379012 s, 138 MB/s PASS 32c (9s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 19:46:43 (1788738403) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (1s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 19:46:44 (1788738404) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (1s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 19:46:45 (1788738405) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (1s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 19:46:46 (1788738406) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (1s) == sanityn test 34: no lock timeout under IO ============= 19:46:47 (1788738407) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg231-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9b9e10447000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b9e10447000.ost_server_uuid in IDLE state after 0 sec oleg231-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b9e10447000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b9e10447000.ost_server_uuid in FULL state after 0 sec oleg231-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9b9e10447000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b9e10447000.ost_server_uuid in IDLE state after 0 sec oleg231-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b9e10447000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b9e10447000.ost_server_uuid in FULL state after 0 sec oleg231-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9b9e10447000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b9e10447000.ost_server_uuid in IDLE state after 0 sec oleg231-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b9e10447000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b9e10447000.ost_server_uuid in IDLE state after 0 sec PASS 34 (91s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 19:48:18 (1788738498) fail_loc=0x80000317 Race attempt 0 Wait for 50445 50580 for 60 sec... total: 4000 open/close in 8.62 seconds: 464.30 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.202.131@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.202.131@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (64s) == sanityn test 36: handle ESTALE/open-unlink correctly == 19:49:22 (1788738562) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg231-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.382305 s, 137 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.460368 s, 114 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.392781 s, 133 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.405174 s, 129 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7495680) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.402924 s, 130 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.469025 s, 112 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.376707 s, 139 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.401819 s, 130 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.428406 s, 122 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.439418 s, 119 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.458381 s, 114 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7497728) after_dd(7446528) after(7497728) PASS 36 (233s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 19:53:15 (1788738795) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 - create 9260 (time 1788738807.29 total 10.00 last 925.90) total: 10000 create in 10.77 seconds: 928.09 ops/second PASS 37 (15s) == sanityn test 39a: file mtime does not change after rename ========================================================== 19:53:30 (1788738810) repeat after cancel_lru_locks PASS 39a (2s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 19:53:32 (1788738812) repeat after cancel_lru_locks PASS 39b (3s) == sanityn test 39c: check truncate mtime update ================================================================================ 19:53:36 (1788738816) repeat after cancel_lru_locks PASS 39c (3s) == sanityn test 39d: sync write should update mtime ====== 19:53:39 (1788738819) fail_loc=0x411 fail_loc=0 PASS 39d (2s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 19:53:41 (1788738821) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (9s) == sanityn test 40c: pdirops: link and others ======================================================================== 19:53:50 (1788738830) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (8s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 19:53:58 (1788738838) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (8s) == sanityn test 40e: pdirops: rename and others ======================================================================== 19:54:06 (1788738846) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (7s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 19:54:14 (1788738854) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (5s) == sanityn test 41b: pdirops: create vs create ======================================================================== 19:54:19 (1788738859) open: File exists Conflict PASS 41b (5s) == sanityn test 41c: pdirops: create vs link ======================================================================== 19:54:24 (1788738864) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (5s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 19:54:29 (1788738869) Conflict PASS 41d (6s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 19:54:35 (1788738875) Conflict PASS 41e (5s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 19:54:40 (1788738880) Conflict PASS 41f (5s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 19:54:45 (1788738885) Conflict PASS 41g (5s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 19:54:50 (1788738890) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (6s) == sanityn test 41i: reint_open: create vs create ======== 19:54:56 (1788738896) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1107s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 20:13:23 (1788740003) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (16s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 20:13:39 (1788740019) open: File exists Conflict PASS 42b (16s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 20:13:55 (1788740035) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (13s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 20:14:09 (1788740049) Conflict PASS 42d (14s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 20:14:23 (1788740063) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (16s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 20:14:39 (1788740079) Conflict PASS 42f (13s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 20:14:52 (1788740092) Conflict PASS 42g (12s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 20:15:04 (1788740104) Conflict PASS 42h (11s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 20:15:15 (1788740115) PASS 43a (62s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 20:16:17 (1788740177) Conflict PASS 43b (17s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 20:16:34 (1788740194) Conflict PASS 43c (12s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 20:16:46 (1788740206) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (12s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 20:16:58 (1788740218) Conflict PASS 43e (20s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 20:17:18 (1788740238) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (14s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 20:17:33 (1788740253) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (12s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 20:17:44 (1788740264) Conflict PASS 43h (10s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 20:17:55 (1788740275) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (2s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 20:17:57 (1788740277) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (111s) == sanityn test 43k: unlink vs create ==================== 20:19:48 (1788740388) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1206s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 20:39:54 (1788741594) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Sun Sep 6 20:40:01 EDT 2026 PASS 44a (12s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 20:40:06 (1788741606) open: File exists Conflict PASS 44b (14s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 20:40:20 (1788741620) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (15s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 20:40:35 (1788741635) Conflict PASS 44d (15s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 20:40:50 (1788741650) Conflict PASS 44e (15s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 20:41:05 (1788741665) Conflict PASS 44f (16s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 20:41:22 (1788741682) Conflict PASS 44g (16s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 20:41:38 (1788741698) Conflict PASS 44h (12s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 20:41:50 (1788741710) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (3s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 20:41:54 (1788741714) PASS 45a (119s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 20:43:53 (1788741833) Conflict PASS 45b (14s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 20:44:07 (1788741847) Conflict PASS 45c (13s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 20:44:20 (1788741860) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (12s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 20:44:32 (1788741872) Conflict PASS 45e (14s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 20:44:46 (1788741886) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (14s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 20:45:00 (1788741900) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (14s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 20:45:15 (1788741915) Conflict PASS 45h (14s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 20:45:29 (1788741929) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 20:45:32 (1788741932) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1134s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 21:04:26 (1788743066) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (18s) == sanityn test 46b: pdirops: link vs create ======================================================================== 21:04:44 (1788743084) open: File exists Conflict PASS 46b (20s) == sanityn test 46c: pdirops: link vs link ======================================================================== 21:05:04 (1788743104) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (15s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 21:05:19 (1788743119) Conflict PASS 46d (16s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 21:05:36 (1788743136) Conflict PASS 46e (14s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 21:05:50 (1788743150) Conflict PASS 46f (12s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 21:06:02 (1788743162) Conflict PASS 46g (13s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 21:06:16 (1788743176) Conflict PASS 46h (13s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 21:06:29 (1788743189) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (4s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 21:06:33 (1788743193) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 21:06:37 (1788743197) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (4s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 21:06:41 (1788743201) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (6s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 21:06:48 (1788743208) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (5s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 21:06:53 (1788743213) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (7s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 21:07:01 (1788743221) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (7s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 21:07:08 (1788743228) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (5s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 21:07:13 (1788743233) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.143807 s, 71.2 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (15s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 21:07:28 (1788743248) 0+1 records in 0+1 records out 158 bytes copied, 0.0447315 s, 3.5 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (219390) - No such process PASS 51a (12s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 21:07:40 (1788743260) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0150474 s, 68.1 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189679 s, 5.5 MB/s 3145728 PASS 51b (21s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 21:08:02 (1788743282) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0353895 s, 28.9 kB/s PASS 51c (14s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 21:08:15 (1788743295) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796384 s, 13.2 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (8s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 21:08:23 (1788743303) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 read lease(1) has applied. PASS 51e (8s) == sanityn test 54: rename locking ======================= 21:08:31 (1788743311) llite.lustre-ffff9b9e10447000.xattr_cache=1 llite.lustre-ffff9b9e20121800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 21:08:33.000000000 -0400 Modify: 2026-09-06 21:08:33.000000000 -0400 Change: 2026-09-06 21:08:33.000000000 -0400 Birth: 2026-09-06 21:08:33.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 21:08:39.000000000 -0400 Modify: 2026-09-06 21:08:40.000000000 -0400 Change: 2026-09-06 21:08:40.000000000 -0400 Birth: 2026-09-06 21:08:39.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 21:08:46.000000000 -0400 Modify: 2026-09-06 21:08:46.000000000 -0400 Change: 2026-09-06 21:08:46.000000000 -0400 Birth: 2026-09-06 21:08:45.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 21:08:46.000000000 -0400 Modify: 2026-09-06 21:08:47.000000000 -0400 Change: 2026-09-06 21:08:47.000000000 -0400 Birth: 2026-09-06 21:08:45.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 21:08:52.000000000 -0400 Modify: 2026-09-06 21:08:53.000000000 -0400 Change: 2026-09-06 21:08:53.000000000 -0400 Birth: 2026-09-06 21:08:52.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 21:08:52.000000000 -0400 Modify: 2026-09-06 21:08:53.000000000 -0400 Change: 2026-09-06 21:08:53.000000000 -0400 Birth: 2026-09-06 21:08:52.000000000 -0400 PASS 54 (35s) == sanityn test 55a: rename vs unlink target dir ========= 21:09:06 (1788743346) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (16s) == sanityn test 55b: rename vs unlink source dir ========= 21:09:22 (1788743362) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (14s) == sanityn test 55c: rename vs unlink orphan target dir == 21:09:36 (1788743376) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (19s) == sanityn test 55d: rename file vs link ================= 21:09:55 (1788743395) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (15s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 21:10:10 (1788743410) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (3s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 21:10:14 (1788743414) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (20s) == sanityn test 55g: rename: race with trylock =========== 21:10:34 (1788743434) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (20s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 21:10:54 (1788743454) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788743456 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.455515 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (142s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 21:13:16 (1788743596) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788743602 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 0.946312 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (145s) == sanityn test 60: Verify data_version behaviour ======== 21:15:41 (1788743741) PASS 60 (12s) == sanityn test 70a: cd directory && rm directory ======== 21:15:53 (1788743753) PASS 70a (12s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 21:16:05 (1788743765) PASS 70b (12s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 21:16:17 (1788743777) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00761951 s, 8.6 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (5s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 21:16:22 (1788743782) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0147049 s, 4.5 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (6s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 21:16:28 (1788743788) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (5s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 21:16:33 (1788743793) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0213408 s, 3.1 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (6s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 21:16:39 (1788743799) llite.lustre-ffff9b9e10447000.xattr_cache=1 llite.lustre-ffff9b9e20121800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (10s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 21:16:49 (1788743809) llite.lustre-ffff9b9e10447000.xattr_cache=1 llite.lustre-ffff9b9e20121800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (9s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 21:16:58 (1788743818) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0219876 s, 46.6 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0260352 s, 39.3 kB/s 232221: get lock2 232223: get lock1 232221: done 232221: get lock1 232223: done 232223: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 232223: cannot get lock2: Resource deadlock avoided 232223: exit rc=0 232221: done 232221: exit rc=0 PASS 74 (14s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 21:17:12 (1788743832) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0634744 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568307 s, 18.5 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232804 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (12s) == sanityn test 76: Verify MDT open_files listing ======== 21:17:24 (1788743844) opening files: ........ closing files: ........ PASS 76 (76s) == sanityn test 77a: check FIFO NRS policy =============== 21:18:40 (1788743920) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.486342 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078901 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164445 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233256 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102082 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186723 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121209 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265639 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120028 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255525 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241231 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282047 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154746 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333972 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116947 s, 9.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137749 s, 7.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364863 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334152 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051001 s, 20.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136648 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521941 s, 20.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145315 s, 72.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112039 s, 93.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900752 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127242 s, 82.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171248 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147638 s, 71.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124479 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753862 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18476 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472461 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044734 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452235 s, 23.2 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (12s) == sanityn test 77b: check CRR-N NRS policy ============== 21:18:52 (1788743932) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.815312 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871444 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14899 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166918 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299427 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307548 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304186 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.511951 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34451 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.615983 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417281 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.405114 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411885 s, 2.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.648237 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.572161 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.483674 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.686631 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138397 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258422 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143079 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151543 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111747 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108144 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143458 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105478 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146558 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00566934 s, 185 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00998513 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207775 s, 50.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609296 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197314 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00759628 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330416 s, 31.7 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.827889 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556854 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10277 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737535 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110881 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127947 s, 8.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791933 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17186 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143737 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939039 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138286 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205972 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205274 s, 5.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166209 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212212 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299749 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369163 s, 2.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201022 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00778085 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263617 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153719 s, 68.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246644 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210792 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260259 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296452 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260744 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160676 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282341 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615795 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241702 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213226 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260456 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081136 s, 12.9 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (19s) == sanityn test 77c: check ORR NRS policy ================ 21:19:12 (1788743952) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.476285 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660462 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864686 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12951 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128329 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103043 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170837 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167244 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160884 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24327 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20734 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454163 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353303 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284778 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359537 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370564 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.473708 s, 2.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102109 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10499 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520854 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554769 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588373 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474582 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478596 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550339 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101428 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104971 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647329 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658492 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085956 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952598 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805848 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174221 s, 6.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg231-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg231-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.597973 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118414 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189731 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193302 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26718 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237984 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276683 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385714 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363543 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.465977 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274349 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270732 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381582 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381292 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.564993 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40799 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409595 s, 2.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667194 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085256 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104779 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801529 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623483 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780167 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119306 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722292 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045045 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124771 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812021 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727731 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495071 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641077 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855711 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073398 s, 14.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.544869 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967704 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124052 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754035 s, 13.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261402 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179282 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259731 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143122 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317163 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29575 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28391 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200907 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343618 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225307 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290184 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283635 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124684 s, 8.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634943 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733347 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717912 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152756 s, 68.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131031 s, 80.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908305 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0029421 s, 356 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117724 s, 89.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00299603 s, 350 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152969 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112835 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048288 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642199 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188887 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113061 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272924 s, 38.4 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg231-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg231-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (20s) == sanityn test 77d: check TRR nrs policy ================ 21:19:31 (1788743971) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.572928 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101598 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119702 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849518 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162514 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11756 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267631 s, 3.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248898 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190825 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101109 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320647 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379652 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3058 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369495 s, 2.8 MB/s1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.44041 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399259 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138303 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104259 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860889 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768363 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117182 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106158 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108506 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12523 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905046 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128828 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10028 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147558 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176158 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131708 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120032 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127142 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160992 s, 6.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.487649 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940526 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145183 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223806 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241435 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289084 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238026 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336298 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252806 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307952 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34489 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286302 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.454624 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.495813 s, 2.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411497 s, 2.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370639 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262238 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509392 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164738 s, 63.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515128 s, 20.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830881 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832414 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139575 s, 75.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915098 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935992 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00678215 s, 155 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213019 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209602 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19582 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222046 s, 4.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208931 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341365 s, 30.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216003 s, 4.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.803107 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989969 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082703 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208535 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288072 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314293 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221833 s, 4.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344138 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238455 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313026 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.483835 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35083 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324129 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301531 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.505301 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313112 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186044 s, 5.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473496 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653627 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567616 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460623 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108428 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830237 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662645 s, 15.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806955 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790552 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497408 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431891 s, 24.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442379 s, 23.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104866 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00982251 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111115 s, 94.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594021 s, 17.7 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (21s) == sanityn test 77e: check TBF NID nrs policy ============ 21:19:52 (1788743992) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.202.31@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.740064 s, 22.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230009 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20235 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343106 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219043 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348077 s, 3.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326854 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370116 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321233 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309809 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13564 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.457758 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.491764 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.485217 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177031 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.415188 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132496 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493524 s, 21.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787288 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251921 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857175 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271011 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653109 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056279 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742867 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579168 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085761 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00495711 s, 212 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234662 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825042 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959883 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00768972 s, 136 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372773 s, 28.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.717028 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100413 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734123 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104963 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140509 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228434 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236189 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260096 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276163 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292009 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303386 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306845 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329973 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329008 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377653 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419018 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402979 s, 2.6 MB/sldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00618704 s, 169 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00552408 s, 190 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802014 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00237808 s, 441 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00375082 s, 280 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00247593 s, 424 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116324 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111905 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962498 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17468 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173667 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119946 s, 87.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170528 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085023 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639935 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373831 s, 28.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg231-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg231-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.618929 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105991 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106998 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128859 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213473 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188738 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155035 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758994 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191878 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183695 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307837 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28822 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234866 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26552 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230542 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288276 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205286 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00842392 s, 124 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441714 s, 23.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150769 s, 69.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253372 s, 41.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410584 s, 25.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029892 s, 35.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00541535 s, 194 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00886136 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00547144 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00390063 s, 269 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00390011 s, 269 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132925 s, 78.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00966275 s, 109 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00734533 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00637763 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00759068 s, 138 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.05126 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108658 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108589 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832364 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131392 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224369 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214336 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197426 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137714 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258971 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214373 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281617 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289618 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162115 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248577 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.424028 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.449887 s, 2.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474513 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675258 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012485 s, 84.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393937 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269279 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391578 s, 26.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753752 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409901 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324887 s, 32.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366268 s, 28.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457343 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615696 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745037 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536144 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056515 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120953 s, 86.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg231-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg231-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (29s) == sanityn test 77f: check TBF JobID nrs policy ========== 21:20:21 (1788744021) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg231-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.69705 s, 24.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122858 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158734 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236115 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22323 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301185 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340293 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315203 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357585 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418802 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.523093 s, 2.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3946 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.51512 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399244 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385363 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.473037 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.59567 s, 1.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147021 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260584 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431479 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077632 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469774 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504084 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730022 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843021 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702406 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113887 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114808 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120211 s, 8.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143058 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145214 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201003 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00766972 s, 137 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg231-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg231-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg231-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.684889 s, 24.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387057 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102751 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13115 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175693 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20829 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175208 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150585 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338914 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310413 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254012 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216616 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315289 s, 3.3 MB/sldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35217 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365353 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.461784 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35824 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246265 s, 42.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379544 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139554 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121552 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122942 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13018 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133003 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711719 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145746 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996047 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968662 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139332 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108168 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152766 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204941 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133763 s, 7.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg231-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg231-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg231-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.904438 s, 18.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174979 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934173 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170212 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297324 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153711 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113822 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264169 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347594 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132959 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318161 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.526323 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.647724 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.611251 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.736171 s, 1.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.586068 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.688832 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744869 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664181 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764227 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859356 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123721 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19775 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160353 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10901 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110835 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120305 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125798 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135403 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150755 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18637 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165738 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184258 s, 5.7 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg231-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg231-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg231-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.873964 s, 19.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148648 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222866 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175963 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2095 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260423 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225071 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.487805 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.395035 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.452852 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.461341 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320817 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.480767 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.524318 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.480112 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365986 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.586011 s, 1.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128652 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116789 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960653 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511976 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108695 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125345 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589716 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719337 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756826 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107103 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127356 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18314 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152612 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179874 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141091 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212695 s, 4.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=2 removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (35s) == sanityn test 77g: Change TBF type directly ============ 21:20:57 (1788744057) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (19s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 21:21:16 (1788744076) oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg231-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg231-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg231-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg231-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 PASS 77h (18s) == sanityn test 77i: Change rank of TBF rule ============= 21:21:35 (1788744095) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (36s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 21:22:11 (1788744131) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.756063 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943554 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118128 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101154 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243357 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243266 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262965 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337051 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346916 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32917 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.473788 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422049 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.448077 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403285 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.430745 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.428286 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.444923 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22636 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211789 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212563 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234704 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243371 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259275 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298908 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322466 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248716 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31302 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305271 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28749 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275373 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326771 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350901 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369862 s, 2.8 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.05091 s, 14.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6523 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (66s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 21:23:17 (1788744197) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg231-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg231-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg231-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg231-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg231-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27839 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122068 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122899 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168277 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684048 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101333 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145019 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214767 s, 4.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185972 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305635 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281975 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193094 s, 5.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310548 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305795 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162267 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22994 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365313 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436826 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651929 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072316 s, 14.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799339 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528262 s, 19.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723233 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536493 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721515 s, 14.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804769 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677047 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111466 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732502 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00867963 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119578 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962994 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094099 s, 11.1 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg231-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg231-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8577 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9732 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg231-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg231-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg231-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg231-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg231-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.32003 s, 12.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152929 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172845 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13341 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154165 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220345 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161277 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260952 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263187 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154877 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215847 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195766 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165535 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375374 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387811 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387536 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.446761 s, 2.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849812 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862217 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797709 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082949 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975068 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917033 s, 11.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877092 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111717 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110252 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131784 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131846 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11572 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110169 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127064 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107381 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116005 s, 9.0 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg231-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg231-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0551 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.018 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (159s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 21:25:56 (1788744356) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg231-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg231-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg231-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg231-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg231-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.4553 s, 11.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134606 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148855 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182661 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216011 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259696 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26956 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185661 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215019 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300222 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106466 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406436 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401682 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.432845 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233823 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215543 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.49798 s, 2.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959457 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101838 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949027 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106266 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112116 s, 9.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848635 s, 12.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119874 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117012 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147819 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13889 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154594 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147584 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141394 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116345 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146295 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163635 s, 6.4 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg231-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg231-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0658 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9775 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg231-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg231-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg231-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg231-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg231-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.32331 s, 12.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839108 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783579 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125847 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117105 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12254 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155596 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203465 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115854 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232655 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222731 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130508 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238053 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205729 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263373 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268325 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285511 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06511 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578848 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728385 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00768773 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087854 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.006975 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213752 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115061 s, 91.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736422 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870588 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321252 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106128 s, 98.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149477 s, 70.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105341 s, 99.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00564343 s, 186 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261493 s, 40.1 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9954 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0072 s, 5.2 MB/s Read runtime is 22 s, speed is 4.545454 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (157s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 21:28:33 (1788744513) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg231-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.974752 s, 17.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140375 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151661 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109827 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217931 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287732 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253353 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25639 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247649 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311696 s, 3.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300691 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379626 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.504083 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421644 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.498403 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.477148 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.522263 s, 2.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506463 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126709 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640005 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125735 s, 83.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12001 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602183 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608678 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113652 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11156 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131135 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157764 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132365 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339466 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0058788 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117869 s, 89.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104981 s, 99.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg231-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg231-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.82807 s, 13.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97585 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.31@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.31@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.800273 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856903 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753031 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103742 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966339 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202754 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730311 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121465 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523112 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237797 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316613 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34321 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382218 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306639 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273769 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471207 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.484419 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126285 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156298 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132124 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143356 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148132 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144993 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146518 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15646 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207271 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170718 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184663 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214576 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197993 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195532 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215599 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220352 s, 4.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=4 removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg231-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg231-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.53346 s, 12.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0924 s, 10.4 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.31@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg231-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.569283 s, 29.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425534 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07458 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963105 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175337 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198822 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229841 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177813 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301327 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311493 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248111 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356312 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264659 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357648 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323471 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375286 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302245 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428194 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695238 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626559 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801644 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644333 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947333 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922389 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718751 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921983 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131087 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128563 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138413 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094274 s, 11.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117486 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150503 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124211 s, 8.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg231-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg231-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.32842 s, 14.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.4013 s, 19.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg231-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.610145 s, 27.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112401 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627669 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151649 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254238 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215645 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328735 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351411 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331787 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356152 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27823 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386645 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362719 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418318 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363288 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.430108 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.487332 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735844 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671265 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044222 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411102 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302808 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024115 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074153 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855571 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619846 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797618 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176743 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748631 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188934 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861767 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840918 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010866 s, 96.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg231-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg231-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.7227 s, 8.9 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94161 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.5509 s, 16.0 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99228 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg231-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.04188 s, 16.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533239 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106364 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841267 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471866 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116419 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667883 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198969 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545703 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229006 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384814 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373816 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297123 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296321 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468097 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291421 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394285 s, 26.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07158 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361695 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739562 s, 14.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968039 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579045 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129634 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108312 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151588 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778327 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149212 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121507 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104041 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966211 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104417 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166515 s, 6.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8265 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6526 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg231-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.602406 s, 27.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874253 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118803 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160944 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188093 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20124 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26895 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262283 s, 4.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24379 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255084 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281601 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250783 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239004 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330499 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361079 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346223 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319061 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929538 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443242 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139624 s, 75.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581671 s, 18.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539228 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061171 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681626 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810672 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117988 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914726 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911394 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306939 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234257 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125331 s, 83.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533801 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166499 s, 63.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1417 s, 10.3 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89685 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg231-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.887076 s, 18.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175535 s, 6.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104331 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127089 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190777 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218996 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178578 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222987 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204534 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.438639 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198291 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298872 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327142 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.526055 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.413523 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.604177 s, 1.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111786 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229117 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114951 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304716 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056932 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061544 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647751 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113966 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484554 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548423 s, 19.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145026 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605058 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675812 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123266 s, 85.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274852 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925736 s, 11.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.6586 s, 9.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0252 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.2111 s, 8.6 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99674 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (444s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 21:35:57 (1788744957) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg231-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.651553 s, 25.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584266 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925039 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189541 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244315 s, 4.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214057 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292599 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340494 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286972 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342328 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255814 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347111 s, 3.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281642 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379352 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307152 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37299 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.449324 s, 2.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108449 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061624 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122986 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125108 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124908 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761083 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107526 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1094 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580993 s, 18.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714444 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136759 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108494 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739053 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791715 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846142 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106806 s, 9.8 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg231-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg231-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.54387 s, 12.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.037 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (57s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 21:36:54 (1788745014) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.31@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.31@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.04974 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06405 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828771 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196975 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180713 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222926 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266865 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26105 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257231 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335988 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349559 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360022 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316913 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305724 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399973 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352693 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426278 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159134 s, 6.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17379 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19355 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148065 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121939 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171513 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116574 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175629 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132234 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163792 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162652 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162616 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194557 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224784 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17416 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2138 s, 4.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=2 removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.017 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1288 s, 10.4 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (59s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 21:37:53 (1788745073) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.31@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg231-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.822243 s, 20.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15017 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166484 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17938 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226668 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285348 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333169 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375539 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393303 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337716 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403437 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.42496 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39407 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351069 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468015 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.470891 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.483334 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571075 s, 18.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126642 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148707 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449588 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155838 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10698 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574919 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124308 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143714 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103268 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164675 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681121 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117357 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899792 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111074 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171664 s, 6.1 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.59822 s, 10.9 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.70202 s, 18.4 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (50s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 21:38:43 (1788745123) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg231-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.886914 s, 18.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721352 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441354 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664453 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145116 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190614 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807026 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831434 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200236 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125179 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158221 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279199 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285713 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203475 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210718 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191318 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252446 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560216 s, 18.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595179 s, 17.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101663 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122885 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134705 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778949 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617789 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811546 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583014 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611251 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639744 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131935 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759734 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104707 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084384 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127664 s, 8.2 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg231-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg231-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.0456 s, 8.0 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99672 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.79903 s, 11.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97619 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (104s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 21:40:27 (1788745227) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg231-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.04672 s, 16.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927005 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884447 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142245 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162543 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179689 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16293 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350728 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262233 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360341 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298591 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249936 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308022 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425419 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331775 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320125 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34209 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119581 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446626 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947006 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645646 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707461 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550378 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872293 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806057 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591535 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910597 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126717 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863706 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114451 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114147 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111777 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130766 s, 8.0 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8878 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0068 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (76s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 21:41:43 (1788745303) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg231-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.624356 s, 26.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942792 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154853 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130367 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175321 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225906 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202519 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268112 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300227 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380196 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367566 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32972 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26769 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406835 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398663 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.434706 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.490901 s, 2.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e10447000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9b9e10447000.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610805 s, 17.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554754 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841373 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636821 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468855 s, 22.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048057 s, 21.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427253 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841051 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574173 s, 18.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468121 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050585 s, 20.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515428 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569569 s, 18.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068873 s, 15.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101359 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068066 s, 15.4 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.04799 s, 17.3 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89727 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg231-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.728269 s, 23.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933664 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150378 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112261 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113698 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113613 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262925 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274331 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136478 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316866 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202782 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31494 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349873 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289707 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305998 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224953 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23508 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429524 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633101 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470421 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391161 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449732 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394678 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727204 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743345 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631853 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654462 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877284 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896556 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844418 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814183 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809529 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100609 s, 10.4 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7467 s, 9.8 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93922 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.6469 s, 9.0 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98332 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (147s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 21:44:10 (1788745450) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg231-client.virtnet /mnt/lustre (opts:) Stopping client oleg231-client.virtnet /mnt/lustre opts: Stopping clients: oleg231-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg231-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg231-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg231-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.31,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg231-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.31,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg231-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.31,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69702/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64-retry1 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.31,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg231-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing load_modules_local oleg231-server: kmemleak disabled oleg231-server: Loading modules from /home/green/git/lustre-release/lustre oleg231-server: detected 4 online CPUs by sysfs oleg231-server: MODOPTS_LIBCFS= oleg231-server: Force libcfs to create 2 CPU partitions oleg231-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre Starting client oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre Started clients oleg231-client.virtnet: 192.168.202.131@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre2 Starting client oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre2 Started clients oleg231-client.virtnet: 192.168.202.131@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b9e02fa8800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9b9e087e9800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b9e02fa8800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b9e087e9800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.906199 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491471 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611927 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600583 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239297 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212234 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265889 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273365 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297277 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311882 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320079 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327637 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316201 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374987 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339433 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408262 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.442944 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329816 s, 31.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489788 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375407 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389677 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750941 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573794 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624993 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483301 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486595 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529004 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057926 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912049 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918499 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060779 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666152 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713016 s, 14.7 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg231-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg231-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.3729 s, 14.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.50313 s, 19.1 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.485104 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833204 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030537 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130166 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182864 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138869 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188827 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217086 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247702 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341354 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189789 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293399 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388455 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314408 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437786 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334632 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.459974 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e02fa8800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9b9e02fa8800.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100197 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123837 s, 84.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340353 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519578 s, 20.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1022 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110945 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110594 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122489 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137516 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146566 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144581 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226152 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147134 s, 7.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9b9e02fa8800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153439 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164702 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171014 s, 6.1 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg231-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg231-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.86113 s, 11.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8458 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (225s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 21:47:55 (1788745675) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg231-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (17s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 21:48:13 (1788745693) On MGS 192.168.202.131, active = nodemap.active=1 On MGS 192.168.202.131, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.522333 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581113 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07883 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056085 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625306 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594099 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137345 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156101 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219324 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234213 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251923 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243199 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272164 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318028 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316755 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324832 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343868 s, 3.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9b9e02fa8800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888084 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076118 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928153 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115405 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110554 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137288 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116837 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13884 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140154 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143741 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166168 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154952 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169548 s, 6.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180169 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172819 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220091 s, 4.8 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.80874 s, 11.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.23069 s, 20.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.202.131, TBF.id = On MGS 192.168.202.131, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.62479 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628668 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613764 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936248 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783896 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729405 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972073 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116511 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847585 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434669 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841691 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110298 s, 9.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9b9e02fa8800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171761 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26105 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313806 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228127 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00255034 s, 411 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103275 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04587 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344454 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696927 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781923 s, 13.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e02fa8800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9b9e02fa8800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00952756 s, 110 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877641 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118937 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00736874 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150419 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0072047 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295682 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169916 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00803429 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108901 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00948622 s, 111 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.16583 s, 11.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.27183 s, 32.0 MB/s Read runtime is 5 s, speed is 20.000000 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg231-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg231-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 On MGS 192.168.202.131, active = nodemap.active=1 On MGS 192.168.202.131, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.784124 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246448 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620736 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681534 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041322 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510376 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180679 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189101 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190069 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216488 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244678 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267851 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297732 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303653 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346696 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344673 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36476 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17405 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187562 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13105 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186169 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948067 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970561 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602512 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21436 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11187 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129508 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202694 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150102 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22119 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151418 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168104 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177137 s, 5.9 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg231-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg231-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.7487 s, 8.9 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94297 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.202.131, TBF.id = On MGS 192.168.202.131, active = nodemap.active=0 PASS 77kj (187s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 21:51:21 (1788745881) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (11s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 21:51:32 (1788745892) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg231-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.6867 s, 284 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg231-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.113 s, 261 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (59s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 21:52:31 (1788745951) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.350651 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493721 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751586 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473587 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802645 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193497 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719378 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225741 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276343 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284901 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869735 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341004 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248701 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273314 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227111 s, 4.6 MB/s1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9b9e02fa8800.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25177 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416429 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935515 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280073 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312283 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372707 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397579 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534182 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532337 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595281 s, 17.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105807 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649782 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072086 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819862 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151018 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869507 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905878 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102212 s, 10.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg231-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg231-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.0119 s, 9.5 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.19325 s, 20.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.622494 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695229 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10115 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206488 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247337 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203604 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28426 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1787 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32914 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231253 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229153 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191371 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316954 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363573 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143862 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.445466 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387435 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9b9e02fa8800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9b9e02fa8800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117503 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122863 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120957 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149816 s, 7.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123026 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147613 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149311 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156696 s, 6.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178321 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153029 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135156 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162378 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163817 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199731 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163295 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173276 s, 6.1 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg231-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg231-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.32185 s, 11.2 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.4439 s, 19.3 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (97s) == sanityn test 77o: Changing rank should not panic ====== 21:54:08 (1788746048) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (13s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 21:54:21 (1788746061) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (116s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 21:56:17 (1788746177) mds.MDS.mdt.nrs_policies=tbf oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg231-client: oleg231-server: ssh exited with exit code 36 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 oleg231-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg231-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (38s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 21:56:55 (1788746215) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273653 kill md_thread xargs PASS 77r (61s) == sanityn test 77s: Check TBF LRU shrinker ============== 21:57:56 (1788746276) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (67s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 21:59:03 (1788746343) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (11s) == sanityn test 79: xattr: intent error ================== 21:59:14 (1788746354) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935685 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 21:59:17.000000000 -0400 Modify: 2026-09-06 21:59:17.000000000 -0400 Change: 2026-09-06 21:59:17.000000000 -0400 Birth: 2026-09-06 21:59:16.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (21s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 21:59:35 (1788746375) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 21:59:38 (1788746378) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (4s) == sanityn test 81a: rename and stat under striped directory ========================================================== 21:59:43 (1788746383) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 21:59:46 (1788746386) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (4s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 21:59:50 (1788746390) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 21:59:54 (1788746394) - open/close 742 (time 1788746411.69 total 10.00 last 74.20) - open/close 1475 (time 1788746421.70 total 20.01 last 73.22) - open/close 2204 (time 1788746431.71 total 30.02 last 72.85) total: 2500 open/close in 34.34 seconds: 72.80 ops/second snapshot_time 1788746522.310062518 secs.nsecs start_time 1788746397.794834360 secs.nsecs elapsed_time 124.515228158 secs.nsecs open 2505 samples [usecs] 138 16127 3933413 10739066727 close 2505 samples [usecs] 148 6350 902237 519119303 mknod 2501 samples [usecs] 540 16018 3429963 8346592465 unlink 2500 samples [usecs] 696 104314 4774988 41770618644 mkdir 2 samples [usecs] 1052 3436 4488 12912800 rmdir 2 samples [usecs] 1053 11924 12977 143290585 rename 2500 samples [usecs] 364 51427 16618672 234167845660 getattr 3760 samples [usecs] 7 10629 241931 424866899 parallel_rename_file 2500 samples [usecs] 364 51427 16618672 234167845660 crossdir_rename 2500 samples [usecs] 364 51427 16618672 234167845660 readpage 18 samples [usecs] 5629 41890 336887 7988009479 PASS 81d (134s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 22:02:08 (1788746528) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7630 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7630 PASS 82 (8s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 22:02:16 (1788746536) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (5s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 22:02:21 (1788746541) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 22:02:34 (1788746554) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788746555 Finishing test test1 at 1788746561 PASS 85 (13s) == sanityn test 90: open/create and unlink striped directory ========================================================== 22:02:47 (1788746567) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (4s) == sanityn test 91: chmod and unlink striped directory === 22:02:51 (1788746571) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 22:02:56 (1788746576) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 22:02:59 (1788746579) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5833 0x16c9 0x240000400 1 5731 0x1663 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5732 0x1664 0x280000400 0 5834 0x16ca 0x240000400 rc1=0 and rc2=0 PASS 93 (18s) == sanityn test 94: signal vs CP callback race =========== 22:03:17 (1788746597) osc.lustre-OST0000-osc-ffff9b9e02fa8800.idle_timeout=0 osc.lustre-OST0000-osc-ffff9b9e087e9800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9b9e02fa8800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9b9e087e9800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.195319 s, 41.9 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.03639 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 290522 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0463821 s, 88.3 kB/s osc.lustre-OST0000-osc-ffff9b9e02fa8800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9b9e087e9800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b9e02fa8800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b9e087e9800.idle_timeout=debug PASS 94 (16s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 22:03:34 (1788746614) llite.lustre-ffff9b9e02fa8800.fast_read=0 llite.lustre-ffff9b9e087e9800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0656991 s, 31.9 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9b9e02fa8800.fast_read=1 llite.lustre-ffff9b9e087e9800.fast_read=1 PASS 95a (21s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 22:03:55 (1788746635) llite.lustre-ffff9b9e02fa8800.fast_read=0 llite.lustre-ffff9b9e087e9800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00768437 s, 1.6 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9b9e02fa8800.fast_read=1 llite.lustre-ffff9b9e087e9800.fast_read=1 PASS 95b (13s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 22:04:08 (1788746648) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 22:04:12 (1788746652) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0120601 s, 340 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (8s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 22:04:20 (1788746660) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.203422 s, 10.3 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (7s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 22:04:28 (1788746668) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.205885 s, 10.2 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (8s) == sanityn test 100e: DoM: read on open and file size ==== 22:04:35 (1788746675) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00557568 s, 2.2 MB/s PASS 100e (7s) == sanityn test 101a: Discard DoM data on unlink ========= 22:04:42 (1788746682) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0169359 s, 242 kB/s PASS 101a (7s) == sanityn test 101b: Discard DoM data on rename ========= 22:04:49 (1788746689) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0070519 s, 581 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 22:04:56 (1788746696) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0127912 s, 320 kB/s PASS 101c (8s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 22:05:05 (1788746705) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 66 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf66:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (8s) == sanityn test 103: Test size correctness with lockahead ========================================================== 22:05:12 (1788746712) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788746715 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788746718 Finishing test test23 at 1788746718 PASS 103 (13s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 22:05:26 (1788746726) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 22:05:29 (1788746729) fail_loc=0x416 PASS 105 (28s) == sanityn test 106a: Verify the btime via statx() ======= 22:05:58 (1788746758) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 22:06:01 (1788746761) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0202387 s, 50.6 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938221 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 22:06:03.000000000 -0400 Modify: 2026-09-06 22:06:03.000000000 -0400 Change: 2026-09-06 22:06:03.000000000 -0400 Birth: 2026-09-06 22:06:03.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938221 -rw-r--r-- 1 0 0 1788746763 1788746763 1788746763 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938221 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 22:06:03.000000000 -0400 Modify: 2026-09-06 22:06:03.000000000 -0400 Change: 2026-09-06 22:06:03.000000000 -0400 Birth: 2026-09-06 22:06:03.000000000 -0400 1788746763 1024 1 PASS 106b (8s) == sanityn test 106c: Verify statx attributes mask ======= 22:06:09 (1788746769) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 22:06:18 (1788746778) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.338196 s, 31.0 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5739 0x166b 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 First grouplock blocks second one PASS 107a (10s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 22:06:28 (1788746788) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.373228 s, 28.1 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5843 0x16d3 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (14s) == sanityn test 108a: lseek: parallel updates ============ 22:06:41 (1788746801) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0176872 s, 565 kB/s llite.lustre-ffff9b9e02fa8800.hybrid_io=0 llite.lustre-ffff9b9e087e9800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.56968 s, 1.8 MB/s PASS 108a (13s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 22:06:54 (1788746814) 192.168.202.131@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg231-client.virtnet /mnt/lustre (opts:) 192.168.202.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg231-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre2 Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre 192.168.202.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.131@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg231-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg231-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing load_modules_local oleg231-server: kmemleak disabled oleg231-server: Loading modules from /home/green/git/lustre-release/lustre oleg231-server: detected 4 online CPUs by sysfs oleg231-server: MODOPTS_LIBCFS= oleg231-server: Force libcfs to create 2 CPU partitions oleg231-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre2 Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre 192.168.202.131@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg231-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg231-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing load_modules_local oleg231-server: kmemleak disabled oleg231-server: Loading modules from /home/green/git/lustre-release/lustre oleg231-server: detected 4 online CPUs by sysfs oleg231-server: MODOPTS_LIBCFS= oleg231-server: Force libcfs to create 2 CPU partitions oleg231-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre2 PASS 109 (68s) == sanityn test 110: do not grant another lock on resend ========================================================== 22:08:02 (1788746882) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 22:08:04.000000000 -0400 Modify: 2026-09-06 22:08:04.000000000 -0400 Change: 2026-09-06 22:08:04.000000000 -0400 Birth: 2026-09-06 22:08:03.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg231-client.virtnet oleg231-client.virtnet PASS 110 (71s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 22:09:13 (1788746953) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (3s) == sanityn test 112: update max-inherit in default LMV === 22:09:16 (1788746956) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 22:09:20 (1788746960) lustre-OST0000-osc-ffff9b9e10442800 active. lustre-OST0001-osc-ffff9b9e10442800 active. lustre-OST0000-osc-ffff9b9e10442800 active. lustre-OST0001-osc-ffff9b9e10442800 active. MGC192.168.202.131@tcp active. PASS 113 (6s) == sanityn test 114: implicit default LMV inherit ======== 22:09:26 (1788746966) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (3s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 22:09:29 (1788746969) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 22:09:32 (1788746972) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (4s) == sanityn test 121: trunc append race =================== 22:09:36 (1788746976) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0410179 s, 499 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0785426 s, 261 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (9s) == sanityn test 122: directory size is consistent across mounts ========================================================== 22:09:45 (1788746985) PASS 122 (9s) == sanityn test 200: service remains healthy while able to process request ========================================================== 22:09:55 (1788746995) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=8 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2458 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2306 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2647 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2534 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.229 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.306 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.221 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2465 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2559 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.242 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2244 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2407 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3896 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4215 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4038 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4912 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg231-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9b9e0601a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b9e0601a000.ost_server_uuid in IDLE state after 0 sec PASS 200 (81s) cleanup: ====================================================== == sanityn test complete, duration 9175 sec ============== 22:11:17 (1788747077) === sanityn: start cleanup 22:11:19 (1788747079) === Stopping clients: oleg231-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg231-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 22:15:24 (1788747324) ===