-----============= acceptance-small: sanityn ============----- Fri Oct 2 18:02:46 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITYN_ONLY='' SANITYN_EXCEPT='' oleg305-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 56b skipping tests SLOW=no: 33a === sanityn: start setup 18:03:13 (1790978593) === Starting client oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 Mount client oleg305-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 Started clients oleg305-client.virtnet: 192.168.203.105@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg305-client.virtnet: executing check_config_client /mnt/lustre oleg305-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg305-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9e3d8586d800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9e3d91ae8800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e3d8586d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e3d91ae8800.idle_timeout=debug disable quota as required oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 18:03:54 (1790978634) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047982 s, 21.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7617] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 18:03:57 (1790978637) PASS 0 (10s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 18:04:07 (1790978647) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (9s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 18:04:16 (1790978656) -rw-r--r-- 1 root root 0 Oct 2 18:04 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (9s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 18:04:25 (1790978665) -rw-r--r-- 1 root root 0 Oct 2 18:04 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (7s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 18:04:32 (1790978672) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (8s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 18:04:40 (1790978680) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (8s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 18:04:48 (1790978688) total 1 -rwxrwxrwx 1 root root 0 Oct 2 18:04 f2a -rwxrwxrwx 1 root root 0 Oct 2 18:04 f2b total 1 -rwxrwxrwx 1 root root 0 Oct 2 18:04 f2a -rwxrwxrwx 1 root root 0 Oct 2 18:04 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0246747 s, 20.7 kB/s PASS 2e (8s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 18:04:56 (1790978696) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 18:05:01 (1790978701) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.405783 s, 5.2 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (8s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 18:05:08 (1790978708) PASS 3 (8s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 18:05:16 (1790978716) PASS 4 (9s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 18:05:25 (1790978725) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (8s) == sanityn test 6: remove of open file on other node ============================================================================ 18:05:33 (1790978733) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 18:05:41 (1790978741) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (7s) == sanityn test 8: remove of open special file on other node ==================================================================== 18:05:48 (1790978748) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (8s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 18:05:56 (1790978756) PASS 9a (9s) == sanityn test 9b: append to striped sparse file ======== 18:06:05 (1790978765) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556925 s, 18.8 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0740997 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (8s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 18:06:13 (1790978773) 1+0 records in 1+0 records out 1 byte copied, 0.00462395 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0236795 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0154275 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0290456 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00879088 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0147027 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0127403 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0227618 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0156867 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0353743 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.013188 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.014227 s, 0.1 kB/s PASS 10a (9s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 18:06:22 (1790978782) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0334038 s, 92.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.124236 s, 33.0 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000319752 s, 9.6 MB/s PASS 10b (8s) == sanityn test 11: execution of file opened for write should return error ============================================================== 18:06:30 (1790978790) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7617 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 304: /mnt/lustre2/d11/f: Text file busy PASS 11 (9s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 18:06:39 (1790978799) warning: '-runtime' deprecated, use '-t runtime' instead - link 278 (time 1790978812.33 total 10.02 last 27.73) - link 522 (time 1790978822.35 total 20.05 last 24.34) - link 758 (time 1790978832.37 total 30.06 last 23.57) - link 1040 (time 1790978842.38 total 40.07 last 28.18) - link 1283 (time 1790978852.38 total 50.08 last 24.28) - link 1526 (time 1790978862.39 total 60.08 last 24.28) - link 1789 (time 1790978872.40 total 70.10 last 26.27) - link 2069 (time 1790978882.43 total 80.13 last 27.91) - link 2345 (time 1790978892.44 total 90.13 last 27.58) total: 2606 link in 100.01 seconds: 26.06 ops/second using seed 2602988102 running for 100 seconds - stat 3995 (time 1790978814 ; total 11 ; last 11) - stat 8206 (time 1790978825 ; total 22 ; last 11) - stat 10000 (time 1790978829 ; total 26 ; last 4) - stat 14454 (time 1790978840 ; total 37 ; last 11) - stat 18810 (time 1790978851 ; total 48 ; last 11) - stat 20000 (time 1790978853 ; total 50 ; last 2) - stat 22431 (time 1790978864 ; total 61 ; last 11) - stat 25171 (time 1790978875 ; total 72 ; last 11) - stat 27576 (time 1790978886 ; total 83 ; last 11) - stat 30000 (time 1790978895 ; total 92 ; last 9) total: 31254 stats in 100 seconds: 312.540009 stats/second - unlinked 0 (time 1790978853 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2606) error: No such file or directory total: 2605 unlinks in 87 seconds: 29.942530 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21442) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21445) - No such process PASS 12 (147s) == sanityn test 13: test directory page revocation ======= 18:09:06 (1790978946) f13 f13 PASS 13 (10s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 18:09:16 (1790978956) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7617 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 339: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 22630 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (8s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 18:09:24 (1790978964) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 23209 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (8s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 18:09:32 (1790978972) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 354: 23788 Terminated $DIR1/$tdir/sleep 60 PASS 14b (9s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 18:09:41 (1790978981) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 366: 24370 Terminated $DIR1/$tdir/sleep 60 PASS 14c (8s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 18:09:49 (1790978989) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 377: 24952 Terminated $DIR1/$tdir/sleep 60 PASS 14d (10s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 18:09:59 (1790978999) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (58s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 18:10:57 (1790979057) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.754757 s, 11.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.481174 s, 17.4 MB/s Chance of close/open is 1 in 50 Seed set to 9063 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read skipping zero length punch hole skipping zero size read 000007[0] 1790979065.255774 write 0x0e7946 thru 0x0e9ae1 (0x219c bytes) truncating to largest ever: 0x5805fb 000008[1] 1790979065.259503 trunc from 0x0e9ae2 to 0x5805fa (0x496b19 bytes) 000009[0] 1790979065.310162 read 0x47a279 thru 0x48116a (0x6ef2 bytes) 000011[1] 1790979065.367643 read 0x1ac2bf thru 0x1b86a3 (0xc3e5 bytes) 000014[0] 1790979065.442542 mapwrite 0x4ad2a9 thru 0x4b06e5 (0x343d bytes) 000015[1] 1790979065.616061 read 0x1e06c5 thru 0x1e50f5 (0x4a31 bytes) 000016[0] 1790979065.648117 punch from 0x1e6258 to 0x1e71d6 (0xf7f bytes) 000018[0] 1790979065.697476 mapwrite 0x06fcd3 thru 0x072903 (0x2c31 bytes) 000020[0] 1790979065.843258 mapread 0x3e23e3 thru 0x3e2832 (0x450 bytes) 000022[1] 1790979065.907271 write 0x459a2a thru 0x4634d1 (0x9aa8 bytes) 000023[1] 1790979065.988251 mapread 0x3f66ba thru 0x3ffc5b (0x95a2 bytes) 000025[0] 1790979066.112197 write 0x4856c0 thru 0x48e370 (0x8cb1 bytes) 000026[1] 1790979066.212732 mapread 0x0c2c16 thru 0x0ca6a3 (0x7a8e bytes) 000027[0] 1790979066.243703 punch from 0x51fbda to 0x5282ae (0x86d5 bytes) 000029[1] 1790979066.275660 write 0x9f9750 thru 0x9fffff (0x68b0 bytes) 000030[0] 1790979066.324019 write 0x971c43 thru 0x972ce2 (0x10a0 bytes) 000032[1] 1790979066.420261 punch from 0x70fdf9 to 0x715f0f (0x6117 bytes) 000033[0] 1790979066.471671 mapread 0x0a22c4 thru 0x0a9504 (0x7241 bytes) 000034[1] 1790979066.529345 punch from 0x61f655 to 0x620476 (0xe22 bytes) 000036[1] 1790979066.579652 read 0x007e77 thru 0x008eb3 (0x103d bytes) truncating to largest ever: 0x7ce6e6 000038[1] 1790979066.674100 trunc from 0xa00000 to 0x7ce6e5 (0x23191a bytes) 000039[1] 1790979066.761543 mapwrite 0x4e4901 thru 0x4e95e0 (0x4ce0 bytes) 000044[0] 1790979067.022527 write 0x48361e thru 0x492931 (0xf314 bytes) 000047[1] 1790979067.125512 punch from 0x1eec5c to 0x1f19d4 (0x2d79 bytes) 000050[1] 1790979067.198772 write 0x8e0743 thru 0x8e7f2f (0x77ed bytes) 000051[0] 1790979067.220499 trunc from 0x8e7f30 to 0x7c661e (0x121911 bytes) 000054[0] 1790979067.333198 write 0x7b2f58 thru 0x7b77e6 (0x488f bytes) 000059[1] 1790979067.537455 write 0x86022e thru 0x868a71 (0x8844 bytes) 000060[0] 1790979067.580837 write 0x0406bb thru 0x04784b (0x7191 bytes) 000061[0] 1790979067.620849 punch from 0x277390 to 0x27ee54 (0x7ac5 bytes) 000062[1] 1790979067.660659 punch from 0x1a14a4 to 0x1b0e6f (0xf9cc bytes) 000068[0] 1790979067.812050 read 0x7bb69c thru 0x7c7a9c (0xc401 bytes) 000070[0] 1790979067.914255 write 0x0c70d5 thru 0x0c72b8 (0x1e4 bytes) 000073[0] 1790979067.995283 punch from 0x5f576e to 0x5fdb23 (0x83b6 bytes) truncating to largest ever: 0x9bb850 000074[0] 1790979068.037723 trunc from 0x868a72 to 0x9bb84f (0x152dde bytes) 000075[1] 1790979068.116138 write 0x7deca0 thru 0x7dee46 (0x1a7 bytes) 000076[1] 1790979068.198045 trunc from 0x9bb850 to 0x3eb280 (0x5d05cf bytes) 000077[1] 1790979068.281242 punch from 0x252583 to 0x2525ef (0x6d bytes) 000078[0] 1790979068.305799 trunc from 0x3eb281 to 0x5fba9d (0x21081d bytes) 000080[0] 1790979068.434242 write 0x88d489 thru 0x88ea93 (0x160b bytes) 000083[1] 1790979068.478482 punch from 0x063d12 to 0x06ce27 (0x9116 bytes) 000084[1] 1790979068.567747 mapwrite 0x2c5b5c thru 0x2d06e8 (0xab8d bytes) 000085[1] 1790979068.909295 write 0x6060e0 thru 0x612c7d (0xcb9e bytes) 000086[0] 1790979068.968690 trunc from 0x88ea94 to 0x123d0b (0x76ad88 bytes) 000088[0] 1790979069.236242 trunc from 0x123d0c to 0x3d2e51 (0x2af146 bytes) 000089[0] 1790979069.323698 mapread 0x282304 thru 0x28875e (0x645b bytes) 000091[0] 1790979069.495494 mapwrite 0x2948cc thru 0x2963bd (0x1af2 bytes) 000092[0] 1790979069.610490 read 0x16fc10 thru 0x1785cd (0x89be bytes) 000094[0] 1790979069.759416 mapwrite 0x06b75a thru 0x075277 (0x9b1e bytes) 000098[0] 1790979069.926564 punch from 0x09eca7 to 0x0aab13 (0xbe6d bytes) 000099[1] 1790979069.997102 write 0x67ebce thru 0x683b5f (0x4f92 bytes) 000100[1] 1790979070.072969 write 0x168f10 thru 0x16d554 (0x4645 bytes) 000101[1] 1790979070.129364 trunc from 0x683b60 to 0x28f973 (0x3f41ec bytes) 000102[0] 1790979070.224436 trunc from 0x28f974 to 0x8cfaf0 (0x64017d bytes) 000104[1] 1790979070.428095 read 0x2123dd thru 0x220164 (0xdd88 bytes) 000105[1] 1790979070.498695 read 0x8cdd60 thru 0x8cfaf0 (0x1d91 bytes) 000106[1] 1790979070.530273 trunc from 0x8cfaf1 to 0x359987 (0x576169 bytes) 000108[0] 1790979070.703540 read 0x04dee2 thru 0x04ff3b (0x205a bytes) 000109[0] 1790979070.845814 mapread 0x0df6a6 thru 0x0ea3ac (0xad07 bytes) 000112[1] 1790979070.989148 mapread 0x24745b thru 0x24af03 (0x3aa9 bytes) 000113[0] 1790979071.041742 trunc from 0x359988 to 0x21c8c9 (0x13d0be bytes) 000116[0] 1790979071.228405 write 0x0ea9f0 thru 0x0ed40a (0x2a1b bytes) 000117[1] 1790979071.247154 mapwrite 0x73c432 thru 0x740214 (0x3de3 bytes) 000118[0] 1790979071.531676 trunc from 0x740215 to 0x4c31c6 (0x27d04e bytes) 000119[1] 1790979071.573090 trunc from 0x4c31c7 to 0x9aa85b (0x4e7695 bytes) 000125[0] 1790979071.782431 trunc from 0x9aa85c to 0x3bbf85 (0x5ee8d6 bytes) 000127[1] 1790979072.009167 write 0x2f179d thru 0x2feb27 (0xd38b bytes) 000128[0] 1790979072.080761 trunc from 0x3bbf86 to 0x34f2c5 (0x6ccc0 bytes) 000129[1] 1790979072.213726 punch from 0x34b76b to 0x34f2c5 (0x3b5b bytes) 000131[1] 1790979072.344423 trunc from 0x34f2c6 to 0x6b1c4b (0x362986 bytes) 000132[1] 1790979072.545378 punch from 0x48b62f to 0x491573 (0x5f45 bytes) 000136[1] 1790979072.747751 mapwrite 0x58dff0 thru 0x597ee6 (0x9ef7 bytes) 000137[1] 1790979072.836590 mapread 0x3c2a0a thru 0x3ca692 (0x7c89 bytes) 000141[0] 1790979072.893341 punch from 0x61accd to 0x620a45 (0x5d79 bytes) 000143[0] 1790979072.951585 mapwrite 0x46eae2 thru 0x46ff2a (0x1449 bytes) 000144[0] 1790979073.072758 mapwrite 0x70d294 thru 0x712a34 (0x57a1 bytes) 000145[0] 1790979073.297631 punch from 0x6fe672 to 0x706bc3 (0x8552 bytes) 000146[1] 1790979073.309274 punch from 0x05fa9e to 0x05ff6e (0x4d1 bytes) 000147[1] 1790979073.385946 trunc from 0x712a35 to 0x266b1c (0x4abf18 bytes) 000151[1] 1790979073.601986 trunc from 0x266b1d to 0x66b5b9 (0x404a9d bytes) 000152[0] 1790979073.682490 punch from 0x473269 to 0x47e2a0 (0xb038 bytes) 000153[1] 1790979073.789308 write 0x3c7cb7 thru 0x3cc6ae (0x49f8 bytes) 000154[0] 1790979073.858351 write 0x53d620 thru 0x544404 (0x6de5 bytes) 000155[0] 1790979073.921809 mapread 0x6359ea thru 0x637b94 (0x21ab bytes) 000156[0] 1790979074.005473 mapwrite 0x265354 thru 0x2742e0 (0xef8d bytes) 000157[0] 1790979074.476643 mapwrite 0x42b408 thru 0x42de76 (0x2a6f bytes) 000158[1] 1790979074.659596 trunc from 0x66b5ba to 0x411fdf (0x2595da bytes) 000160[0] 1790979074.792826 punch from 0x3ba2a8 to 0x3c47e9 (0xa542 bytes) 000161[0] 1790979074.858714 mapread 0x160f69 thru 0x16dde4 (0xce7c bytes) 000164[1] 1790979075.062361 mapwrite 0x7150da thru 0x722fd6 (0xdefd bytes) 000165[0] 1790979075.315045 write 0x2026b9 thru 0x210589 (0xded1 bytes) 000166[1] 1790979075.389575 write 0x1de912 thru 0x1e3bbe (0x52ad bytes) 000167[1] 1790979075.536169 punch from 0x65d87e to 0x66d389 (0xfb0c bytes) 000169[1] 1790979075.637206 mapwrite 0x4d7c28 thru 0x4dd3d0 (0x57a9 bytes) 000170[0] 1790979075.791698 write 0x890f7c thru 0x896894 (0x5919 bytes) 000171[0] 1790979075.843915 trunc from 0x896895 to 0x482e80 (0x413a14 bytes) 000173[0] 1790979075.934690 punch from 0x1b4ec8 to 0x1b5809 (0x942 bytes) 000175[0] 1790979075.939985 read 0x4030f1 thru 0x40c5a9 (0x94b9 bytes) 000177[0] 1790979076.000341 write 0x40c1e8 thru 0x40d0b7 (0xed0 bytes) 000178[0] 1790979076.029760 read 0x46af16 thru 0x46f00b (0x40f6 bytes) 000179[0] 1790979076.042670 mapread 0x44b60c thru 0x456637 (0xb02c bytes) 000186[0] 1790979076.139024 trunc from 0x482e81 to 0x186024 (0x2fce5c bytes) 000187[1] 1790979076.162321 read 0x0cbd07 thru 0x0dbb16 (0xfe10 bytes) 000188[1] 1790979076.251036 read 0x02871b thru 0x02fa67 (0x734d bytes) 000190[0] 1790979076.386468 punch from 0x0f3f90 to 0x0f8e84 (0x4ef5 bytes) 000194[0] 1790979076.447316 punch from 0x16b209 to 0x178971 (0xd769 bytes) 000195[1] 1790979076.454662 punch from 0x14ab89 to 0x14b3d5 (0x84d bytes) 000196[1] 1790979076.505430 punch from 0x0baade to 0x0c8892 (0xddb5 bytes) 000199[0] 1790979076.595599 punch from 0x0d0ae7 to 0x0d55c7 (0x4ae1 bytes) 000200[0] 1790979076.652277 trunc from 0x186025 to 0x031c14 (0x154410 bytes) 000202[0] 1790979076.697398 mapwrite 0x9e956c thru 0x9f922d (0xfcc2 bytes) 000203[0] 1790979076.901733 trunc from 0x9f922e to 0x6ff0ae (0x2fa17f bytes) 000204[0] 1790979076.924080 trunc from 0x6ff0af to 0x01f7d2 (0x6df8dc bytes) 000205[1] 1790979077.004714 mapread 0x01ba8f thru 0x01f7d2 (0x3d44 bytes) 000206[0] 1790979077.159136 punch from 0x00eb16 to 0x00fbb6 (0x10a1 bytes) 000207[1] 1790979077.187938 trunc from 0x01f7d3 to 0x309e2d (0x2ea65b bytes) 000209[0] 1790979077.326833 trunc from 0x309e2e to 0x811a81 (0x507c54 bytes) 000210[0] 1790979077.431238 trunc from 0x811a82 to 0x011803 (0x80027e bytes) 000211[1] 1790979077.462264 write 0x84e54f thru 0x85d879 (0xf32b bytes) 000212[1] 1790979077.565237 trunc from 0x85d87a to 0x648fee (0x21488b bytes) 000213[0] 1790979077.616597 mapwrite 0x085869 thru 0x08d78a (0x7f22 bytes) 000215[0] 1790979077.946321 write 0x5819d0 thru 0x58dec8 (0xc4f9 bytes) 000218[0] 1790979078.061707 mapread 0x333f9c thru 0x33ed72 (0xadd7 bytes) 000219[1] 1790979078.153347 mapwrite 0x885a13 thru 0x88d07b (0x7669 bytes) 000220[1] 1790979078.597092 mapwrite 0x90c8de thru 0x90d033 (0x756 bytes) 000221[1] 1790979078.795187 mapread 0x32f7b6 thru 0x331a58 (0x22a3 bytes) 000222[1] 1790979078.858756 trunc from 0x90d034 to 0x781759 (0x18b8da bytes) 000225[0] 1790979079.028985 write 0x3f196e thru 0x3fc08d (0xa720 bytes) 000227[0] 1790979079.183200 trunc from 0x78175a to 0x921c5a (0x1a0501 bytes) 000229[1] 1790979079.378988 write 0x8ce5b3 thru 0x8d125c (0x2caa bytes) 000230[0] 1790979079.500039 trunc from 0x921c5b to 0x4607dc (0x4c147e bytes) 000231[1] 1790979079.596004 punch from 0x14ac21 to 0x154fde (0xa3be bytes) 000233[1] 1790979079.652516 punch from 0x104693 to 0x106ac5 (0x2433 bytes) 000235[1] 1790979079.738395 trunc from 0x4607dd to 0x2b75cc (0x1a9210 bytes) 000237[0] 1790979079.870771 trunc from 0x2b75cd to 0x517175 (0x25fba9 bytes) 000238[0] 1790979079.929545 trunc from 0x517176 to 0x861716 (0x34a5a1 bytes) 000239[0] 1790979079.955968 mapread 0x711369 thru 0x7186dc (0x7374 bytes) 000241[1] 1790979079.979839 punch from 0x52883c to 0x5383e4 (0xfba9 bytes) 000242[1] 1790979080.026440 write 0x898228 thru 0x8a3c87 (0xba60 bytes) 000243[0] 1790979080.098284 trunc from 0x8a3c88 to 0x183ca1 (0x71ffe6 bytes) 000244[0] 1790979080.213214 trunc from 0x183ca2 to 0x6888cd (0x504c2c bytes) 000245[0] 1790979080.239742 read 0x655370 thru 0x65d08e (0x7d1f bytes) 000246[0] 1790979080.289994 punch from 0x3388d4 to 0x343b59 (0xb286 bytes) 000247[0] 1790979080.330794 mapwrite 0x5c4ace thru 0x5c6c2f (0x2162 bytes) 000252[0] 1790979080.525409 read 0x4ddac2 thru 0x4e023c (0x277b bytes) 000255[0] 1790979080.604397 trunc from 0x6888ce to 0x0d934e (0x5af57f bytes) 000257[1] 1790979080.716346 mapwrite 0x8ec0e0 thru 0x8f1360 (0x5281 bytes) 000259[1] 1790979080.941368 punch from 0x49b37e to 0x4a6ead (0xbb30 bytes) 000261[1] 1790979081.000018 trunc from 0x8f1361 to 0x6eeb94 (0x2027cc bytes) 000262[1] 1790979081.028396 read 0x58fa72 thru 0x5925da (0x2b69 bytes) 000263[1] 1790979081.080240 mapread 0x1abc12 thru 0x1af0ce (0x34bd bytes) 000265[1] 1790979081.132851 mapwrite 0x51f88b thru 0x529085 (0x97fb bytes) 000266[0] 1790979081.382318 trunc from 0x6eeb95 to 0x8e5c3f (0x1f70ab bytes) 000268[0] 1790979081.463650 mapread 0x2103df thru 0x21b0f3 (0xad15 bytes) 000270[1] 1790979081.508621 trunc from 0x8e5c40 to 0x712dd1 (0x1d2e6e bytes) 000272[1] 1790979081.584390 write 0x013dc4 thru 0x01f797 (0xb9d4 bytes) 000275[1] 1790979081.626359 mapread 0x438fb8 thru 0x439a93 (0xadc bytes) 000276[1] 1790979081.658758 trunc from 0x712dd2 to 0x8e4046 (0x1d1275 bytes) 000277[0] 1790979081.685405 punch from 0x2d5183 to 0x2e3b4a (0xe9c8 bytes) 000278[1] 1790979081.728278 read 0x115ee3 thru 0x11a77b (0x4899 bytes) 000279[0] 1790979081.765822 punch from 0x83951d to 0x84305f (0x9b43 bytes) 000282[1] 1790979081.870088 read 0x83dcec thru 0x843aa5 (0x5dba bytes) 000283[1] 1790979081.905619 read 0x5e17fd thru 0x5ec460 (0xac64 bytes) 000284[1] 1790979081.927984 punch from 0x2a5a20 to 0x2ab44d (0x5a2e bytes) 000286[1] 1790979082.046332 read 0x77c368 thru 0x7803d1 (0x406a bytes) 000291[1] 1790979082.245174 write 0x230fc7 thru 0x23df53 (0xcf8d bytes) 000293[1] 1790979082.347517 mapread 0x890cb9 thru 0x899da1 (0x90e9 bytes) 000294[1] 1790979082.498725 punch from 0x082997 to 0x08e373 (0xb9dd bytes) 000295[1] 1790979082.564689 punch from 0x6e8ecd to 0x6ecf13 (0x4047 bytes) 000297[0] 1790979082.575818 mapread 0x5612e1 thru 0x56f13f (0xde5f bytes) 000298[1] 1790979082.635957 trunc from 0x8e4047 to 0x1acefc (0x73714a bytes) 000300[1] 1790979082.705419 punch from 0x04cce2 to 0x0531a4 (0x64c3 bytes) 000301[0] 1790979082.729524 punch from 0x0fa849 to 0x109ef4 (0xf6ac bytes) 000302[0] 1790979082.773919 punch from 0x0e3506 to 0x0ee99f (0xb49a bytes) 000303[0] 1790979082.791369 mapread 0x0882dc thru 0x08ce28 (0x4b4d bytes) 000304[0] 1790979082.840395 read 0x02f5d0 thru 0x02f69c (0xcd bytes) 000305[0] 1790979082.933356 punch from 0x068acb to 0x075912 (0xce48 bytes) 000306[1] 1790979082.966387 trunc from 0x1acefd to 0x808334 (0x65b438 bytes) 000308[0] 1790979083.156419 trunc from 0x808335 to 0x3f23b7 (0x415f7d bytes) 000311[0] 1790979083.252291 write 0x99f68b thru 0x9a4e7f (0x57f5 bytes) 000312[0] 1790979083.326912 mapread 0x8e87b0 thru 0x8ed5b7 (0x4e08 bytes) 000313[0] 1790979083.484780 read 0x4a2d5b thru 0x4a63a1 (0x3647 bytes) 000314[0] 1790979083.485401 punch from 0x528823 to 0x53667d (0xde5b bytes) 000315[1] 1790979083.582067 mapwrite 0x5ddd4d thru 0x5ecc38 (0xeeec bytes) 000316[0] 1790979084.229218 write 0x6c7db9 thru 0x6d1ce6 (0x9f2e bytes) 000320[0] 1790979084.340419 write 0x039f0e thru 0x04003d (0x6130 bytes) 000321[1] 1790979084.351809 mapread 0x5a0ab2 thru 0x5af6aa (0xebf9 bytes) 000322[0] 1790979084.425273 write 0x978d73 thru 0x98145c (0x86ea bytes) 000323[0] 1790979084.428211 punch from 0x3a803c to 0x3aefcb (0x6f90 bytes) 000324[1] 1790979084.464429 write 0x58e50a thru 0x59a18d (0xbc84 bytes) 000326[1] 1790979084.593797 punch from 0x37b584 to 0x385986 (0xa403 bytes) 000329[1] 1790979084.691236 punch from 0x9772ef to 0x982fbd (0xbccf bytes) 000331[0] 1790979084.749126 punch from 0x3e70da to 0x3e8dbb (0x1ce2 bytes) 000332[0] 1790979084.804742 write 0x07a46b thru 0x07c2ed (0x1e83 bytes) 000333[0] 1790979084.845530 read 0x0599ba thru 0x068937 (0xef7e bytes) 000338[0] 1790979084.991314 punch from 0x8b63aa to 0x8c0e1f (0xaa76 bytes) 000339[0] 1790979085.029046 mapwrite 0x11dbe3 thru 0x128c09 (0xb027 bytes) 000340[0] 1790979085.147538 punch from 0x1778e2 to 0x1781f4 (0x913 bytes) 000342[1] 1790979085.198946 punch from 0x7a14d0 to 0x7a5f98 (0x4ac9 bytes) 000343[1] 1790979085.231841 mapwrite 0x4cb128 thru 0x4d8be4 (0xdabd bytes) 000344[0] 1790979085.479051 write 0x3f8a7d thru 0x4029c6 (0x9f4a bytes) 000345[0] 1790979085.545592 trunc from 0x9a4e80 to 0x8a2774 (0x10270b bytes) 000346[1] 1790979085.652498 write 0x35ab3a thru 0x36123b (0x6702 bytes) 000347[1] 1790979085.755761 write 0x9129b2 thru 0x91d41d (0xaa6c bytes) 000348[0] 1790979085.786771 mapread 0x3e7cc2 thru 0x3e93a2 (0x16e1 bytes) 000349[0] 1790979085.901270 trunc from 0x91d41e to 0x3d38c6 (0x549b57 bytes) 000351[0] 1790979085.984626 mapwrite 0x5a6184 thru 0x5b107d (0xaefa bytes) 000352[0] 1790979086.103486 read 0x0bcaae thru 0x0c521d (0x8770 bytes) 000353[0] 1790979086.148606 read 0x42f33a thru 0x436fa3 (0x7c6a bytes) 000354[1] 1790979086.184325 trunc from 0x5b107e to 0x08a34b (0x526d32 bytes) 000355[0] 1790979086.243619 punch from 0x06927d to 0x06c0ec (0x2e70 bytes) 000358[1] 1790979086.311550 read 0x035876 thru 0x043e94 (0xe61f bytes) 000359[0] 1790979086.385294 mapwrite 0x1cd70c thru 0x1db9ff (0xe2f4 bytes) 000360[0] 1790979086.568444 punch from 0x062176 to 0x06c398 (0xa223 bytes) 000362[1] 1790979086.632145 punch from 0x08716d to 0x08ae54 (0x3ce8 bytes) 000363[0] 1790979086.681659 punch from 0x032269 to 0x039301 (0x7099 bytes) 000365[0] 1790979086.771324 write 0x739ba3 thru 0x742fef (0x944d bytes) 000367[0] 1790979086.779281 punch from 0x6057c5 to 0x60f518 (0x9d54 bytes) 000371[0] 1790979086.824749 punch from 0x205bc3 to 0x207e3c (0x227a bytes) 000375[1] 1790979086.906022 write 0x23109e thru 0x234820 (0x3783 bytes) 000376[1] 1790979086.936032 mapwrite 0x4da757 thru 0x4e0c77 (0x6521 bytes) 000377[1] 1790979087.304525 punch from 0x6042fa to 0x613bcb (0xf8d2 bytes) 000379[0] 1790979087.314537 mapread 0x485eb5 thru 0x48af83 (0x50cf bytes) 000380[1] 1790979087.405830 punch from 0x677112 to 0x67c4e5 (0x53d4 bytes) 000381[1] 1790979087.480608 punch from 0x592030 to 0x59e174 (0xc145 bytes) 000384[0] 1790979087.598559 mapwrite 0x7caa75 thru 0x7d4242 (0x97ce bytes) 000385[0] 1790979087.834903 mapread 0x77d717 thru 0x789c0f (0xc4f9 bytes) 000389[1] 1790979087.959506 punch from 0x294d94 to 0x2953e1 (0x64e bytes) 000395[1] 1790979088.159910 mapwrite 0x544631 thru 0x54a5ad (0x5f7d bytes) 000398[0] 1790979088.472917 mapwrite 0x280202 thru 0x2870d9 (0x6ed8 bytes) 000399[0] 1790979088.676765 mapwrite 0x27ffa7 thru 0x28d112 (0xd16c bytes) 000400[1] 1790979088.824608 write 0x52f5dd thru 0x5378ee (0x8312 bytes) 000401[1] 1790979088.859644 trunc from 0x7d4243 to 0x048af7 (0x78b74b bytes) 000402[1] 1790979088.916376 read 0x0246bb thru 0x029c08 (0x554e bytes) 000403[1] 1790979088.932049 mapread 0x047f37 thru 0x048af7 (0xbc1 bytes) 000404[1] 1790979088.932737 read 0x0473ce thru 0x048af7 (0x172a bytes) 000405[0] 1790979088.933177 trunc from 0x048af8 to 0x6596a3 (0x610bac bytes) 000406[1] 1790979089.026345 mapwrite 0x0d2aec thru 0x0d5797 (0x2cac bytes) 000407[1] 1790979089.470490 mapread 0x1ed3ec thru 0x1ee35d (0xf72 bytes) 000409[1] 1790979089.559290 mapread 0x13f4a4 thru 0x14b702 (0xc25f bytes) 000410[0] 1790979089.642509 mapread 0x13df13 thru 0x146121 (0x820f bytes) 000411[0] 1790979089.777691 write 0x8512c8 thru 0x858f05 (0x7c3e bytes) 000412[1] 1790979089.831320 trunc from 0x858f06 to 0x078741 (0x7e07c4 bytes) 000413[0] 1790979090.038935 write 0x98311d thru 0x9886c7 (0x55ab bytes) 000414[1] 1790979090.123628 mapread 0x260a23 thru 0x265871 (0x4e4f bytes) 000416[1] 1790979090.314282 mapwrite 0x9ac32a thru 0x9bbd5e (0xfa35 bytes) 000417[0] 1790979090.669322 mapwrite 0x08e40f thru 0x096004 (0x7bf6 bytes) 000420[0] 1790979090.839759 trunc from 0x9bbd5f to 0x8c9e27 (0xf1f37 bytes) 000421[1] 1790979090.892640 punch from 0x1ed0f3 to 0x1f549b (0x83a9 bytes) 000422[1] 1790979090.969767 write 0x9c52ff thru 0x9ce09b (0x8d9d bytes) 000424[0] 1790979091.048723 trunc from 0x9ce09c to 0x353852 (0x67a849 bytes) 000427[1] 1790979091.194804 mapwrite 0x6967fa thru 0x6a2a0d (0xc214 bytes) 000428[1] 1790979091.533608 trunc from 0x6a2a0e to 0x2a8d9b (0x3f9c72 bytes) 000430[0] 1790979091.621092 punch from 0x1318a9 to 0x13e7ae (0xcf06 bytes) 000431[1] 1790979091.674567 mapread 0x1dcd86 thru 0x1e6cb7 (0x9f32 bytes) 000432[1] 1790979091.836104 trunc from 0x2a8d9c to 0x540c04 (0x297e69 bytes) 000434[1] 1790979091.957665 mapwrite 0x785942 thru 0x78e515 (0x8bd4 bytes) 000436[1] 1790979092.256536 read 0x5ad429 thru 0x5b96f8 (0xc2d0 bytes) 000437[0] 1790979092.312831 mapread 0x16caee thru 0x174879 (0x7d8c bytes) 000439[0] 1790979092.443401 write 0x4682d1 thru 0x477380 (0xf0b0 bytes) 000440[1] 1790979092.487281 mapwrite 0x8ce8d3 thru 0x8d7cad (0x93db bytes) 000441[1] 1790979092.635775 punch from 0x53797f to 0x53cbaf (0x5231 bytes) 000442[1] 1790979092.658252 mapwrite 0x64fd8e thru 0x653ec0 (0x4133 bytes) 000443[0] 1790979092.775815 punch from 0x847eb1 to 0x851e8f (0x9fdf bytes) 000444[0] 1790979092.828978 mapwrite 0x45f3d5 thru 0x46c4f6 (0xd122 bytes) 000445[0] 1790979092.986941 mapread 0x85f0c4 thru 0x868845 (0x9782 bytes) 000446[0] 1790979093.064296 trunc from 0x8d7cae to 0x9ab8a3 (0xd3bf6 bytes) 000447[1] 1790979093.154243 punch from 0x5fe0e6 to 0x6021af (0x40ca bytes) 000449[0] 1790979093.248208 punch from 0x5b1cab to 0x5b3d67 (0x20bd bytes) 000450[0] 1790979093.271004 read 0x549729 thru 0x5544d0 (0xada8 bytes) 000451[1] 1790979093.318430 mapread 0x63263b thru 0x641bc7 (0xf58d bytes) 000452[1] 1790979093.403752 punch from 0x3733ed to 0x378f8b (0x5b9f bytes) 000456[0] 1790979093.478525 write 0x32556b thru 0x330b9c (0xb632 bytes) 000457[0] 1790979093.568416 trunc from 0x9ab8a4 to 0x74f70f (0x25c194 bytes) 000460[1] 1790979093.744488 mapread 0x4aebfa thru 0x4bb78d (0xcb94 bytes) 000461[1] 1790979093.892273 punch from 0x3f45eb to 0x3ffcbe (0xb6d4 bytes) truncating to largest ever: 0x9eb682 000462[0] 1790979093.964240 trunc from 0x74f710 to 0x9eb681 (0x29bf72 bytes) 000463[0] 1790979094.068283 read 0x5c09a6 thru 0x5c2aff (0x215a bytes) 000465[1] 1790979094.109391 mapwrite 0x4cd75f thru 0x4d6d8c (0x962e bytes) 000466[1] 1790979094.300607 mapwrite 0x3f9a8e thru 0x406912 (0xce85 bytes) 000467[1] 1790979094.597486 trunc from 0x9eb682 to 0x603678 (0x3e8009 bytes) 000468[1] 1790979094.710931 write 0x536bb9 thru 0x543760 (0xcba8 bytes) 000469[0] 1790979094.793576 mapread 0x0f091c thru 0x0ff425 (0xeb0a bytes) 000470[0] 1790979094.943272 trunc from 0x603679 to 0x930bda (0x32d562 bytes) 000471[1] 1790979095.020986 write 0x311c74 thru 0x31fc92 (0xe01f bytes) 000473[1] 1790979095.089755 trunc from 0x930bdb to 0x923fae (0xcc2c bytes) 000474[0] 1790979095.133439 mapwrite 0x93e782 thru 0x94347d (0x4cfc bytes) 000475[0] 1790979095.267712 read 0x457b0c thru 0x466d77 (0xf26c bytes) 000477[1] 1790979095.348605 mapwrite 0x672fbe thru 0x67f8fc (0xc93f bytes) 000478[1] 1790979095.564985 trunc from 0x94347e to 0x3cf9d9 (0x573aa4 bytes) 000479[0] 1790979095.640379 write 0x60c1f1 thru 0x61bc5d (0xfa6d bytes) 000480[0] 1790979095.688288 read 0x511892 thru 0x515888 (0x3ff7 bytes) 000481[0] 1790979095.757598 mapwrite 0x62b10b thru 0x631cc3 (0x6bb9 bytes) 000482[1] 1790979095.950302 punch from 0x20e0ae to 0x2199b5 (0xb908 bytes) 000483[1] 1790979095.990237 write 0x8216da thru 0x82613b (0x4a62 bytes) 000484[1] 1790979096.018579 write 0x60142b thru 0x607bc9 (0x679f bytes) 000489[0] 1790979096.154221 read 0x0a87bd thru 0x0a9b11 (0x1355 bytes) 000492[1] 1790979096.337797 trunc from 0x82613c to 0x6c95f8 (0x15cb43 bytes) 000498[0] 1790979096.612842 punch from 0x2bc98a to 0x2c163c (0x4cb3 bytes) 000499[1] 1790979096.650287 punch from 0x2d3f1f to 0x2de3ff (0xa4e1 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 9096 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole truncating to largest ever: 0x8db190 000004[0] 1790979097.744942 trunc from 00000000 to 0x8db18f (0x8db190 bytes) 000005[1] 1790979097.810588 punch from 0x89c6cc to 0x89f391 (0x2cc6 bytes) 000006[0] 1790979097.836201 punch from 0x30b4e7 to 0x319805 (0xe31f bytes) 000009[1] 1790979097.880617 read 0x7fc000 thru 0x804a59 (0x8a5a bytes) 000010[1] 1790979097.913764 mapwrite 0x8c2000 thru 0x8c223d (0x23e bytes) 000012[1] 1790979098.049198 punch from 0x5cfa74 to 0x5dc202 (0xc78f bytes) 000013[0] 1790979098.054494 trunc from 0x8db190 to 0x60a33c (0x2d0e53 bytes) 000014[1] 1790979098.131389 write 0x558000 thru 0x55d68c (0x568d bytes) 000015[0] 1790979098.172124 read_OD 0x3fd000 thru 0x407fff (0xb000 bytes) 000017[0] 1790979098.268741 mapread 0x15b000 thru 0x160e49 (0x5e4a bytes) 000018[0] 1790979098.303443 punch from 0x1fb654 to 0x204db0 (0x975d bytes) 000020[1] 1790979098.395676 trunc from 0x60a33d to 0x3361a7 (0x2d4195 bytes) 000021[0] 1790979098.452386 punch from 0x26c154 to 0x27a351 (0xe1fe bytes) 000022[0] 1790979098.498758 punch from 0x1ebbb4 to 0x1f8b0d (0xcf5a bytes) 000023[0] 1790979098.520712 read_OD 0x21d000 thru 0x22bfff (0xf000 bytes) 000025[1] 1790979098.554421 mapread 0x1e3000 thru 0x1f2754 (0xf755 bytes) 000026[0] 1790979098.605693 punch from 0x018e18 to 0x0270b8 (0xe2a1 bytes) 000027[1] 1790979098.644406 mapwrite 0x539000 thru 0x541be0 (0x8be1 bytes) 000032[0] 1790979098.893583 trunc from 0x541be1 to 0x39228e (0x1af952 bytes) 000035[1] 1790979098.979200 read 0x1a0000 thru 0x1ad240 (0xd241 bytes) 000036[0] 1790979099.005255 punch from 0x06d1e2 to 0x07769b (0xa4ba bytes) 000037[1] 1790979099.040705 write 0x808000 thru 0x80fddc (0x7ddd bytes) 000040[0] 1790979099.071772 write_OD 0x61b000 thru 0x621fff (0x7000 bytes) 000043[1] 1790979099.157759 mapwrite 0x541000 thru 0x54c6ff (0xb700 bytes) 000048[0] 1790979099.329465 mapread 0x6b3000 thru 0x6bc496 (0x9497 bytes) 000050[1] 1790979099.577841 trunc from 0x80fddd to 0x18505e (0x68ad7e bytes) 000051[1] 1790979099.627420 punch from 0x14bf27 to 0x14cc4b (0xd25 bytes) 000052[1] 1790979099.649676 trunc from 0x18505f to 0x0aab48 (0xda516 bytes) 000053[0] 1790979099.694148 punch from 0x08fdc7 to 0x09a9e7 (0xac21 bytes) 000060[1] 1790979099.767269 mapread 0x01a000 thru 0x0280ba (0xe0bb bytes) 000061[0] 1790979099.826631 mapread 0x02a000 thru 0x02d574 (0x3575 bytes) 000064[1] 1790979099.870808 punch from 0x01c7f2 to 0x01e5b8 (0x1dc7 bytes) 000066[1] 1790979099.927601 write 0x7a8000 thru 0x7b7bf1 (0xfbf2 bytes) 000067[1] 1790979099.977348 mapwrite 0x75d000 thru 0x76476f (0x7770 bytes) 000072[1] 1790979100.107536 mapwrite 0x553000 thru 0x5590f1 (0x60f2 bytes) 000073[1] 1790979100.275997 trunc from 0x7b7bf2 to 0x0d4727 (0x6e34ca bytes) 000074[1] 1790979100.314580 read 0x023000 thru 0x02ac41 (0x7c42 bytes) 000076[0] 1790979100.431234 trunc from 0x0d4728 to 0x369910 (0x2951e9 bytes) 000079[1] 1790979100.533554 read 0x0ef000 thru 0x0f344e (0x444f bytes) 000081[0] 1790979100.625179 read_OD 0x332000 thru 0x33bfff (0xa000 bytes) 000083[1] 1790979100.632848 write 0x702000 thru 0x711f8a (0xff8b bytes) 000084[0] 1790979100.663387 punch from 0x3fafb7 to 0x404b48 (0x9b92 bytes) 000085[1] 1790979100.693522 mapwrite 0x1db000 thru 0x1dc36b (0x136c bytes) 000086[1] 1790979100.805415 punch from 0x103a9b to 0x10aa8e (0x6ff4 bytes) 000088[0] 1790979100.836839 write_OD 0x6ac000 thru 0x6b7fff (0xc000 bytes) 000089[0] 1790979100.894275 mapread 0x017000 thru 0x01c8b2 (0x58b3 bytes) 000090[0] 1790979101.008198 punch from 0x33e9ef to 0x3406cb (0x1cdd bytes) 000091[0] 1790979101.046504 trunc from 0x711f8b to 0x7b0c4f (0x9ecc5 bytes) 000092[1] 1790979101.074159 mapread 0x039000 thru 0x03c4c7 (0x34c8 bytes) 000093[0] 1790979101.128420 write_OD 0x40b000 thru 0x410fff (0x6000 bytes) 000096[0] 1790979101.185392 trunc from 0x7b0c50 to 0x3b0a4b (0x400204 bytes) 000100[0] 1790979101.274608 read_OD 0x0f6000 thru 0x0fdfff (0x8000 bytes) 000102[0] 1790979101.295709 trunc from 0x3b0a4c to 0x22aa88 (0x185fc3 bytes) 000104[1] 1790979101.348686 write 0x2b8000 thru 0x2c1ad1 (0x9ad2 bytes) 000106[0] 1790979101.368104 punch from 0x2a8f58 to 0x2b1cc7 (0x8d70 bytes) 000109[1] 1790979101.464818 write 0x46d000 thru 0x46d31b (0x31c bytes) 000110[0] 1790979101.487571 write_OD 0x800000 thru 0x808fff (0x9000 bytes) 000111[1] 1790979101.571708 mapread 0x12d000 thru 0x1302c6 (0x32c7 bytes) 000112[1] 1790979101.614117 read 0x6c0000 thru 0x6c90ae (0x90af bytes) 000113[0] 1790979101.641212 write_OD 0x470000 thru 0x479fff (0xa000 bytes) 000114[1] 1790979101.744420 write 0x36a000 thru 0x36ca48 (0x2a49 bytes) 000115[0] 1790979101.772138 punch from 0x0276ca to 0x02a78d (0x30c4 bytes) 000116[1] 1790979101.862640 mapwrite 0x5e0000 thru 0x5e9b99 (0x9b9a bytes) 000117[0] 1790979102.082466 mapread 0x4a4000 thru 0x4b3b98 (0xfb99 bytes) 000118[1] 1790979102.354611 mapread 0x56d000 thru 0x5797a9 (0xc7aa bytes) 000119[0] 1790979102.377062 mapread 0x305000 thru 0x305087 (0x88 bytes) 000120[0] 1790979102.377631 trunc from 0x809000 to 0x2c145e (0x547ba1 bytes) 000121[1] 1790979102.559669 punch from 0x11d9ba to 0x12763e (0x9c85 bytes) 000122[0] 1790979102.585967 punch from 0x158ffb to 0x15d79e (0x47a4 bytes) 000124[0] 1790979102.656968 mapwrite 0x260000 thru 0x2681fd (0x81fe bytes) 000125[0] 1790979102.811794 punch from 0x1739fe to 0x180f10 (0xd513 bytes) 000126[1] 1790979102.816540 punch from 0x193117 to 0x1a2ced (0xfbd7 bytes) 000127[1] 1790979102.862532 mapread 0x2b0000 thru 0x2bceec (0xceed bytes) 000130[0] 1790979102.936270 write_OD 0x1e0000 thru 0x1e4fff (0x5000 bytes) 000131[0] 1790979103.039376 trunc from 0x2c145f to 0x811d33 (0x5508d5 bytes) 000132[0] 1790979103.085065 read_OD 0x608000 thru 0x614fff (0xd000 bytes) 000136[0] 1790979103.119103 trunc from 0x811d34 to 0x193db6 (0x67df7d bytes) 000139[1] 1790979103.175815 read 0x16d000 thru 0x178cfb (0xbcfc bytes) 000141[0] 1790979103.225038 mapwrite 0x1ee000 thru 0x1f9ec3 (0xbec4 bytes) 000142[0] 1790979103.370796 punch from 0x1196d0 to 0x11eabb (0x53ec bytes) 000145[1] 1790979103.426651 read 0x142000 thru 0x148d3d (0x6d3e bytes) 000147[0] 1790979103.537207 trunc from 0x1f9ec4 to 0x18f493 (0x6aa30 bytes) 000148[0] 1790979103.630079 mapwrite 0x3dd000 thru 0x3ebd34 (0xed35 bytes) 000149[0] 1790979103.813463 trunc from 0x3ebd35 to 0x4106eb (0x249b7 bytes) 000153[0] 1790979103.867519 write_OD 0x80e000 thru 0x81bfff (0xe000 bytes) 000154[0] 1790979103.957210 mapwrite 0x5e2000 thru 0x5eec7d (0xcc7e bytes) 000155[1] 1790979104.085071 punch from 0x591703 to 0x59de17 (0xc715 bytes) 000158[0] 1790979104.161463 write_OD 0x3ab000 thru 0x3affff (0x5000 bytes) 000159[0] 1790979104.244143 read_OD 0x147000 thru 0x155fff (0xf000 bytes) 000160[0] 1790979104.264144 write_OD 0x6d1000 thru 0x6d4fff (0x4000 bytes) 000161[1] 1790979104.318427 mapwrite 0x128000 thru 0x13119e (0x919f bytes) 000162[0] 1790979104.530679 write_OD 0x863000 thru 0x86ffff (0xd000 bytes) 000165[0] 1790979104.631442 mapread 0x760000 thru 0x76851c (0x851d bytes) 000167[1] 1790979104.657762 punch from 0x759061 to 0x75d233 (0x41d3 bytes) 000168[1] 1790979104.698993 mapwrite 0x879000 thru 0x88363f (0xa640 bytes) 000169[1] 1790979104.924067 write 0x96e000 thru 0x973bc6 (0x5bc7 bytes) 000170[0] 1790979104.932557 mapwrite 0x517000 thru 0x525d81 (0xed82 bytes) 000171[1] 1790979105.082362 punch from 0x1b163f to 0x1b8857 (0x7219 bytes) 000172[0] 1790979105.107396 trunc from 0x973bc7 to 0x126c11 (0x84cfb5 bytes) 000175[0] 1790979105.264398 mapwrite 0x357000 thru 0x35d220 (0x6221 bytes) 000176[0] 1790979105.542037 mapwrite 0x4a5000 thru 0x4a6831 (0x1832 bytes) 000179[0] 1790979105.767977 punch from 0x47b592 to 0x47b81c (0x28b bytes) 000180[1] 1790979105.781055 trunc from 0x4a6832 to 0x333400 (0x173431 bytes) 000181[1] 1790979105.866677 read 0x039000 thru 0x03eed1 (0x5ed2 bytes) 000182[0] 1790979105.996245 write_OD 0x2e0000 thru 0x2e8fff (0x9000 bytes) 000184[0] 1790979106.143717 read_OD 0x11a000 thru 0x11cfff (0x3000 bytes) 000186[0] 1790979106.158691 trunc from 0x333401 to 0x5ab72c (0x27832c bytes) 000190[1] 1790979106.324537 read 0x18f000 thru 0x19b563 (0xc564 bytes) 000191[0] 1790979106.384125 read_OD 0x0cf000 thru 0x0d9fff (0xb000 bytes) 000192[0] 1790979106.418709 mapread 0x302000 thru 0x30f988 (0xd989 bytes) 000194[0] 1790979106.492403 trunc from 0x5ab72d to 0x58291b (0x28e11 bytes) 000197[0] 1790979106.653432 punch from 0x24217f to 0x24f36f (0xd1f1 bytes) 000198[0] 1790979106.669815 trunc from 0x58291c to 0x68f577 (0x10cc5c bytes) 000199[1] 1790979106.748107 mapwrite 0x612000 thru 0x61ae38 (0x8e39 bytes) 000200[1] 1790979106.977536 read 0x33a000 thru 0x340a61 (0x6a62 bytes) 000203[0] 1790979107.006480 write_OD 0x59f000 thru 0x5a2fff (0x4000 bytes) 000204[0] 1790979107.135223 write_OD 0x8a1000 thru 0x8a2fff (0x2000 bytes) 000206[1] 1790979107.281334 read 0x4a5000 thru 0x4b4b3d (0xfb3e bytes) 000207[0] 1790979107.291106 trunc from 0x8a3000 to 0x1978d6 (0x70b729 bytes) 000210[1] 1790979107.388614 punch from 0x09d4b1 to 0x09f2c8 (0x1e18 bytes) 000211[1] 1790979107.409120 trunc from 0x1978d7 to 0x7ea955 (0x65307f bytes) truncating to largest ever: 0x94175c 000212[1] 1790979107.430522 trunc from 0x7ea956 to 0x94175b (0x156e06 bytes) 000213[1] 1790979107.458226 trunc from 0x94175c to 0x13d869 (0x803ef2 bytes) 000214[0] 1790979107.501155 mapwrite 0x7a5000 thru 0x7a6332 (0x1333 bytes) 000215[1] 1790979107.703761 trunc from 0x7a6333 to 0x4e3f78 (0x2c23ba bytes) 000217[0] 1790979107.832356 punch from 0x248e3a to 0x24e649 (0x5810 bytes) 000218[1] 1790979107.899504 write 0x50b000 thru 0x50b897 (0x898 bytes) 000219[0] 1790979107.972397 trunc from 0x50b898 to 0x3d4ebc (0x1369db bytes) 000221[0] 1790979108.146986 trunc from 0x3d4ebd to 0x7076a0 (0x3327e4 bytes) 000222[1] 1790979108.232013 read 0x0ca000 thru 0x0cbe75 (0x1e76 bytes) 000223[0] 1790979108.303094 write_OD 0x665000 thru 0x673fff (0xf000 bytes) 000224[0] 1790979108.519057 read_OD 0x12e000 thru 0x12efff (0x1000 bytes) 000225[1] 1790979108.539583 mapread 0x696000 thru 0x6a3e54 (0xde55 bytes) 000227[0] 1790979108.623351 mapread 0x411000 thru 0x41c7cc (0xb7cd bytes) 000228[0] 1790979108.698981 read_OD 0x6e0000 thru 0x6e7fff (0x8000 bytes) 000230[0] 1790979108.718908 write_OD 0x6d0000 thru 0x6ddfff (0xe000 bytes) 000231[1] 1790979108.818090 punch from 0x20d4b6 to 0x212343 (0x4e8e bytes) 000232[1] 1790979108.838337 punch from 0x44cf36 to 0x45ce99 (0xff64 bytes) 000233[0] 1790979108.864199 write_OD 0x89d000 thru 0x8a9fff (0xd000 bytes) 000234[0] 1790979108.931456 trunc from 0x8aa000 to 0x2364b4 (0x673b4b bytes) 000235[0] 1790979108.972908 mapread 0x0dd000 thru 0x0e0dfd (0x3dfe bytes) 000236[0] 1790979109.030915 punch from 0x0bc910 to 0x0c5890 (0x8f81 bytes) 000238[1] 1790979109.052583 read 0x215000 thru 0x21778b (0x278c bytes) 000239[0] 1790979109.196375 mapwrite 0x097000 thru 0x09995e (0x295f bytes) 000240[0] 1790979109.464435 trunc from 0x2364b5 to 0x2f7cfb (0xc1847 bytes) 000242[0] 1790979109.588203 mapread 0x22e000 thru 0x233bdb (0x5bdc bytes) 000243[0] 1790979109.674651 write_OD 0x52f000 thru 0x53bfff (0xd000 bytes) 000245[1] 1790979109.858953 punch from 0x41c9b5 to 0x42a71f (0xdd6b bytes) 000246[1] 1790979109.926449 mapread 0x3ef000 thru 0x3f2839 (0x383a bytes) 000247[0] 1790979109.978977 write_OD 0x9ad000 thru 0x9b2fff (0x6000 bytes) 000249[0] 1790979110.178402 punch from 0x222910 to 0x224def (0x24e0 bytes) 000250[0] 1790979110.208097 trunc from 0x9b3000 to 0x1155ab (0x89da54 bytes) 000251[0] 1790979110.275074 read_OD 0x014000 thru 0x01efff (0xb000 bytes) 000252[1] 1790979110.324195 trunc from 0x1155ac to 0x86d6eb (0x758140 bytes) 000253[0] 1790979110.376328 write_OD 0x653000 thru 0x65bfff (0x9000 bytes) 000254[1] 1790979110.516201 punch from 0x78b105 to 0x78f643 (0x453f bytes) 000256[1] 1790979110.641527 mapread 0x22b000 thru 0x23411e (0x911f bytes) 000257[1] 1790979110.689044 punch from 0x3cf8f1 to 0x3da4b9 (0xabc9 bytes) 000260[1] 1790979110.790543 read_OD 0x61a000 thru 0x622fff (0x9000 bytes) 000261[0] 1790979110.813710 read_OD 0x19a000 thru 0x1a3fff (0xa000 bytes) 000262[0] 1790979110.829392 read_OD 0x61d000 thru 0x629fff (0xd000 bytes) 000263[1] 1790979110.842178 punch from 0x7fe250 to 0x80140f (0x31c0 bytes) 000266[1] 1790979110.881115 trunc from 0x86d6ec to 0x5709bd (0x2fcd2e bytes) 000271[0] 1790979110.994383 mapwrite 0x27a000 thru 0x285a94 (0xba95 bytes) 000273[0] 1790979111.234729 read_OD 0x2b5000 thru 0x2b9fff (0x5000 bytes) 000275[1] 1790979111.274061 punch from 0x23e32d to 0x24b830 (0xd504 bytes) 000276[1] 1790979111.301500 read_OD 0x449000 thru 0x44cfff (0x4000 bytes) 000277[1] 1790979111.318186 punch from 0x566586 to 0x5709bd (0xa438 bytes) 000278[1] 1790979111.349373 write_OD 0x64c000 thru 0x656fff (0xb000 bytes) 000281[0] 1790979111.414075 mapwrite 0x634000 thru 0x63946f (0x5470 bytes) 000283[1] 1790979111.501354 trunc from 0x657000 to 0x4e2ec7 (0x174138 bytes) 000290[0] 1790979111.587574 trunc from 0x4e2ec8 to 0x570e90 (0x8dfc9 bytes) 000291[0] 1790979111.682376 mapwrite 0x90e000 thru 0x916250 (0x8251 bytes) 000292[1] 1790979111.801198 write_OD 0x779000 thru 0x781fff (0x9000 bytes) 000297[0] 1790979111.989333 write_OD 0x1e4000 thru 0x1f0fff (0xd000 bytes) 000298[0] 1790979112.120258 write_OD 0x0ae000 thru 0x0b3fff (0x6000 bytes) 000299[0] 1790979112.256103 mapread 0x870000 thru 0x877c2f (0x7c30 bytes) 000300[0] 1790979112.289716 trunc from 0x916251 to 0x0f976c (0x81cae4 bytes) 000302[1] 1790979112.429727 punch from 0x04afc4 to 0x050392 (0x53cf bytes) 000303[0] 1790979112.519333 read_OD 0x08a000 thru 0x090fff (0x7000 bytes) 000304[1] 1790979112.580973 read_OD 0x034000 thru 0x040fff (0xd000 bytes) 000305[0] 1790979112.626006 read_OD 0x0c2000 thru 0x0cffff (0xe000 bytes) 000306[0] 1790979112.651130 read_OD 0x0b3000 thru 0x0b6fff (0x4000 bytes) 000307[0] 1790979112.671991 read_OD 0x099000 thru 0x0a5fff (0xd000 bytes) 000308[1] 1790979112.698416 mapread 0x0b1000 thru 0x0b9c60 (0x8c61 bytes) 000312[1] 1790979112.777270 mapwrite 0x40c000 thru 0x41ab33 (0xeb34 bytes) 000314[1] 1790979113.010699 punch from 0x1bebf2 to 0x1ce495 (0xf8a4 bytes) 000315[0] 1790979113.029559 mapwrite 0x9bc000 thru 0x9ca330 (0xe331 bytes) 000316[0] 1790979113.614687 write_OD 0x345000 thru 0x34cfff (0x8000 bytes) 000317[0] 1790979113.800089 trunc from 0x9ca331 to 0x4424dd (0x587e53 bytes) 000318[0] 1790979113.900479 mapread 0x2ea000 thru 0x2ec102 (0x2103 bytes) 000320[1] 1790979113.944799 mapwrite 0x61d000 thru 0x61ea4e (0x1a4f bytes) 000321[0] 1790979114.143854 read_OD 0x012000 thru 0x014fff (0x3000 bytes) 000322[1] 1790979114.186088 read_OD 0x230000 thru 0x236fff (0x7000 bytes) 000324[0] 1790979114.206719 trunc from 0x61ea4f to 0x08fc8c (0x58edc2 bytes) 000326[0] 1790979114.283123 trunc from 0x08fc8d to 0x3cae8b (0x33b1ff bytes) 000327[1] 1790979114.326032 punch from 0x2c93b0 to 0x2d0c21 (0x7872 bytes) 000329[1] 1790979114.374204 mapwrite 0x8ba000 thru 0x8c0734 (0x6735 bytes) 000330[0] 1790979114.540361 write_OD 0x87c000 thru 0x881fff (0x6000 bytes) 000331[1] 1790979114.637472 punch from 0x58b91b to 0x5932e7 (0x79cd bytes) 000332[1] 1790979114.656197 mapwrite 0x7ee000 thru 0x7eeae6 (0xae7 bytes) 000334[0] 1790979114.783778 mapread 0x258000 thru 0x264a59 (0xca5a bytes) 000336[0] 1790979114.903479 punch from 0x6cb2c4 to 0x6d9bcf (0xe90c bytes) 000337[0] 1790979114.923068 mapread 0x57b000 thru 0x57cff3 (0x1ff4 bytes) 000338[1] 1790979114.966538 trunc from 0x8c0735 to 0x16f271 (0x7514c3 bytes) 000340[1] 1790979115.062485 mapread 0x0d5000 thru 0x0df543 (0xa544 bytes) 000341[0] 1790979115.204066 punch from 0x16c4a3 to 0x16f271 (0x2dcf bytes) 000342[0] 1790979115.287485 trunc from 0x16f272 to 0x84bd4b (0x6dcada bytes) 000343[0] 1790979115.329093 write_OD 0x4e1000 thru 0x4e4fff (0x4000 bytes) 000346[0] 1790979115.490845 read_OD 0x356000 thru 0x357fff (0x2000 bytes) 000348[0] 1790979115.498350 punch from 0x7c8d55 to 0x7cef2a (0x61d6 bytes) 000349[0] 1790979115.531245 read_OD 0x5cb000 thru 0x5d2fff (0x8000 bytes) 000350[1] 1790979115.583168 punch from 0x684b6b to 0x6857d8 (0xc6e bytes) 000353[1] 1790979115.694480 write_OD 0x90d000 thru 0x915fff (0x9000 bytes) 000354[0] 1790979115.906655 trunc from 0x916000 to 0x5a1672 (0x37498d bytes) 000355[0] 1790979115.987093 write_OD 0x957000 thru 0x95ffff (0x9000 bytes) 000356[0] 1790979116.155552 write_OD 0x6c5000 thru 0x6c7fff (0x3000 bytes) 000358[0] 1790979116.225496 trunc from 0x960000 to 0x7ab2a3 (0x1b4d5c bytes) 000360[0] 1790979116.288586 trunc from 0x7ab2a4 to 0x37d812 (0x42da91 bytes) 000362[1] 1790979116.316075 mapwrite 0x9ca000 thru 0x9ceb22 (0x4b23 bytes) 000365[1] 1790979116.570509 trunc from 0x9ceb23 to 0x285a8d (0x749095 bytes) 000367[0] 1790979116.676977 mapread 0x1b8000 thru 0x1bb374 (0x3375 bytes) 000369[1] 1790979116.798541 read_OD 0x201000 thru 0x209fff (0x9000 bytes) 000370[1] 1790979116.828574 read_OD 0x15c000 thru 0x161fff (0x6000 bytes) 000372[0] 1790979116.839929 punch from 0x0ad962 to 0x0af3be (0x1a5d bytes) 000374[1] 1790979116.867729 write_OD 0x62a000 thru 0x62cfff (0x3000 bytes) 000375[0] 1790979116.923300 mapwrite 0x74d000 thru 0x758e3e (0xbe3f bytes) 000376[0] 1790979117.169827 trunc from 0x758e3f to 0x431184 (0x327cba bytes) 000377[1] 1790979117.223680 write_OD 0x3b0000 thru 0x3b3fff (0x4000 bytes) 000378[0] 1790979117.351151 mapwrite 0x214000 thru 0x2163d7 (0x23d8 bytes) 000379[0] 1790979117.508531 write_OD 0x071000 thru 0x07dfff (0xd000 bytes) 000380[1] 1790979117.635662 mapwrite 0x245000 thru 0x248a78 (0x3a79 bytes) 000381[1] 1790979117.858662 punch from 0x215937 to 0x2183e6 (0x2ab0 bytes) 000382[0] 1790979117.864478 mapread 0x401000 thru 0x405abf (0x4ac0 bytes) 000383[1] 1790979117.918789 mapread 0x2eb000 thru 0x2ef1af (0x41b0 bytes) 000384[0] 1790979117.941995 write_OD 0x4a2000 thru 0x4adfff (0xc000 bytes) 000385[1] 1790979118.022687 read_OD 0x11f000 thru 0x121fff (0x3000 bytes) 000386[1] 1790979118.035688 write_OD 0x74f000 thru 0x758fff (0xa000 bytes) 000388[1] 1790979118.148977 punch from 0x302a91 to 0x306c16 (0x4186 bytes) 000390[0] 1790979118.210437 mapread 0x730000 thru 0x7358df (0x58e0 bytes) 000394[1] 1790979118.288111 read_OD 0x029000 thru 0x032fff (0xa000 bytes) 000396[0] 1790979118.300475 punch from 0x53bb1a to 0x53cbe9 (0x10d0 bytes) 000399[0] 1790979118.330450 punch from 0x1adf71 to 0x1b2c3e (0x4cce bytes) 000400[1] 1790979118.353556 mapwrite 0x819000 thru 0x81a769 (0x176a bytes) 000403[0] 1790979118.516752 punch from 0x615cbf to 0x6213f6 (0xb738 bytes) 000405[0] 1790979118.583737 mapread 0x6b4000 thru 0x6beb78 (0xab79 bytes) 000406[0] 1790979118.625959 mapwrite 0x8a6000 thru 0x8ab6cf (0x56d0 bytes) 000407[1] 1790979118.915706 mapread 0x772000 thru 0x778800 (0x6801 bytes) 000409[0] 1790979118.973678 mapwrite 0x046000 thru 0x04d88c (0x788d bytes) 000410[1] 1790979119.083934 trunc from 0x8ab6d0 to 0x86f069 (0x3c666 bytes) 000413[0] 1790979119.126513 mapread 0x632000 thru 0x64040b (0xe40c bytes) 000416[0] 1790979119.194772 mapwrite 0x35d000 thru 0x3602e1 (0x32e2 bytes) 000419[1] 1790979119.350569 mapwrite 0x330000 thru 0x338674 (0x8675 bytes) 000421[0] 1790979119.518452 mapread 0x024000 thru 0x031a66 (0xda67 bytes) 000422[1] 1790979119.570958 write_OD 0x23b000 thru 0x23cfff (0x2000 bytes) 000423[0] 1790979119.631529 mapwrite 0x7b1000 thru 0x7bc975 (0xb976 bytes) 000424[0] 1790979119.774442 trunc from 0x86f06a to 0x5b33e0 (0x2bbc89 bytes) 000425[1] 1790979119.831911 mapwrite 0x4f1000 thru 0x4f20bf (0x10c0 bytes) 000427[0] 1790979120.010338 trunc from 0x5b33e1 to 0x917888 (0x3644a8 bytes) 000431[0] 1790979120.192462 trunc from 0x917889 to 0x073892 (0x8a3ff6 bytes) 000432[0] 1790979120.270750 punch from 0x050e38 to 0x0537ed (0x29b6 bytes) truncating to largest ever: 0x9cca78 000434[1] 1790979120.307566 trunc from 0x073893 to 0x9cca77 (0x9591e5 bytes) 000436[0] 1790979120.395706 trunc from 0x9cca78 to 0x1ba038 (0x812a3f bytes) 000438[1] 1790979120.437967 mapread 0x132000 thru 0x13a35b (0x835c bytes) 000439[1] 1790979120.529444 write_OD 0x034000 thru 0x040fff (0xd000 bytes) 000440[0] 1790979120.617495 mapwrite 0x551000 thru 0x5520f0 (0x10f1 bytes) 000441[0] 1790979120.808555 punch from 0x2d618f to 0x2e4888 (0xe6fa bytes) 000442[1] 1790979120.848356 mapread 0x3d5000 thru 0x3d586d (0x86e bytes) 000445[0] 1790979120.895119 mapwrite 0x3d7000 thru 0x3e4f40 (0xdf41 bytes) 000446[1] 1790979121.076405 read_OD 0x1a5000 thru 0x1abfff (0x7000 bytes) 000448[1] 1790979121.124768 mapread 0x19e000 thru 0x1a7010 (0x9011 bytes) 000449[0] 1790979121.166695 punch from 0x491b06 to 0x4957c8 (0x3cc3 bytes) 000451[0] 1790979121.214658 read_OD 0x3c5000 thru 0x3c9fff (0x5000 bytes) 000452[0] 1790979121.257264 write_OD 0x4f5000 thru 0x4f8fff (0x4000 bytes) 000455[0] 1790979121.331044 trunc from 0x5520f1 to 0x7cb195 (0x2790a5 bytes) 000457[1] 1790979121.453708 mapread 0x38a000 thru 0x390601 (0x6602 bytes) 000460[1] 1790979121.548211 write_OD 0x8f8000 thru 0x8fffff (0x8000 bytes) 000462[0] 1790979121.627037 mapwrite 0x58e000 thru 0x59c5e4 (0xe5e5 bytes) 000464[0] 1790979121.858983 punch from 0x202484 to 0x20eccd (0xc84a bytes) 000465[1] 1790979121.914447 write_OD 0x7bd000 thru 0x7c1fff (0x5000 bytes) 000466[0] 1790979122.077452 trunc from 0x900000 to 0x2ea095 (0x615f6a bytes) 000469[1] 1790979122.172286 trunc from 0x2ea096 to 0x894ca7 (0x5aac12 bytes) 000470[1] 1790979122.212667 write_OD 0x54b000 thru 0x557fff (0xd000 bytes) 000472[1] 1790979122.279590 write_OD 0x442000 thru 0x44efff (0xd000 bytes) 000475[1] 1790979122.366566 punch from 0x22e816 to 0x23bb64 (0xd34f bytes) 000477[1] 1790979122.414066 read_OD 0x6cd000 thru 0x6d0fff (0x4000 bytes) 000481[1] 1790979122.446549 mapwrite 0x2f7000 thru 0x303073 (0xc074 bytes) 000484[1] 1790979122.564573 trunc from 0x894ca8 to 0x07a69e (0x81a609 bytes) 000486[1] 1790979122.630597 write_OD 0x937000 thru 0x93efff (0x8000 bytes) 000487[0] 1790979122.700683 punch from 0x2bad5d to 0x2bc274 (0x1518 bytes) 000491[1] 1790979122.763693 punch from 0x012eca to 0x01448d (0x15c4 bytes) 000492[0] 1790979122.818713 read_OD 0x043000 thru 0x04afff (0x8000 bytes) 000493[1] 1790979122.875397 mapread 0x7c2000 thru 0x7d1ee3 (0xfee4 bytes) 000495[0] 1790979122.949903 trunc from 0x93f000 to 0x4fbccd (0x443332 bytes) 000496[1] 1790979123.039742 mapread 0x368000 thru 0x370b32 (0x8b33 bytes) 000497[0] 1790979123.142697 mapread 0x350000 thru 0x35636c (0x636d bytes) 000499[1] 1790979123.178555 mapread 0x19b000 thru 0x1a1db2 (0x6db3 bytes) 000500[0] 1790979123.216721 punch from 0x042c58 to 0x04c529 (0x98d2 bytes) All operations completed A-OK! PASS 16a (72s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 18:12:09 (1790979129) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.534798 s, 15.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.636603 s, 13.2 MB/s Chance of close/open is 1 in 50 Seed set to 9134 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole skipping zero size read 000006[0] 1790979135.440974 mapwrite 0x2f1adc thru 0x2f3052 (0x1577 bytes) truncating to largest ever: 0x4b979a 000008[0] 1790979135.670710 trunc from 0x2f3053 to 0x4b9799 (0x1c6747 bytes) 000009[0] 1790979135.693664 read 0x1caa73 thru 0x1cbbd8 (0x1166 bytes) 000010[1] 1790979135.729197 write 0x91abb5 thru 0x91c9b2 (0x1dfe bytes) 000015[0] 1790979135.911308 mapread 0x547937 thru 0x548827 (0xef1 bytes) 000016[0] 1790979136.005498 mapread 0x721a2d thru 0x72259c (0xb70 bytes) 000017[0] 1790979136.102571 mapread 0x0ee61a thru 0x0efd8e (0x1775 bytes) 000020[1] 1790979136.225265 write 0x225374 thru 0x226ae6 (0x1773 bytes) 000022[1] 1790979136.390313 write 0x939fb2 thru 0x93a516 (0x565 bytes) 000024[1] 1790979136.471217 read 0x40071c thru 0x40126b (0xb50 bytes) 000025[1] 1790979136.549352 punch from 0x052a17 to 0x0546d0 (0x1cba bytes) truncating to largest ever: 0x7195ad 000026[1] 1790979136.609146 trunc from 0x93a517 to 0x7195ac (0x220f6a bytes) 000029[1] 1790979136.692361 mapread 0x296b48 thru 0x29717a (0x633 bytes) 000032[1] 1790979136.767541 write 0x804976 thru 0x8050ec (0x777 bytes) 000033[0] 1790979136.768755 write 0x4677b1 thru 0x468dd0 (0x1620 bytes) 000036[1] 1790979136.858246 write 0x1c5b48 thru 0x1c642f (0x8e8 bytes) 000037[0] 1790979136.885320 read 0x0b3eef thru 0x0b3f9d (0xaf bytes) 000039[1] 1790979136.933512 mapwrite 0x58a1d9 thru 0x58ae26 (0xc4e bytes) 000040[0] 1790979137.060273 mapwrite 0x223097 thru 0x224c4f (0x1bb9 bytes) 000042[1] 1790979137.184359 punch from 0x15fc4f to 0x160aa0 (0xe52 bytes) 000044[1] 1790979137.227906 trunc from 0x8050ed to 0x33cfaf (0x4c813d bytes) 000045[0] 1790979137.255367 write 0x6c05ed thru 0x6c0c2b (0x63f bytes) 000047[1] 1790979137.310731 mapwrite 0x2f03a1 thru 0x2f0e48 (0xaa8 bytes) 000050[0] 1790979137.423478 mapwrite 0x0910fb thru 0x092339 (0x123f bytes) truncating to largest ever: 0x8037f1 000051[0] 1790979137.516072 trunc from 0x6c0c2c to 0x8037f0 (0x142bc5 bytes) 000056[1] 1790979137.652205 read 0x12563c thru 0x127359 (0x1d1e bytes) 000058[1] 1790979137.692329 trunc from 0x8037f1 to 0x51d533 (0x2e62bd bytes) 000061[0] 1790979137.779462 trunc from 0x51d534 to 0x025572 (0x4f7fc1 bytes) 000062[0] 1790979137.862443 trunc from 0x025573 to 0x6cb36b (0x6a5df9 bytes) 000063[1] 1790979137.924152 read 0x2cb326 thru 0x2cbaeb (0x7c6 bytes) 000065[0] 1790979138.012384 write 0x693f71 thru 0x694eef (0xf7f bytes) 000066[1] 1790979138.058241 trunc from 0x6cb36c to 0x6efad5 (0x2476a bytes) 000067[1] 1790979138.141133 punch from 0x5917c3 to 0x591e13 (0x651 bytes) 000070[0] 1790979138.293319 mapwrite 0x974f33 thru 0x975274 (0x342 bytes) 000071[1] 1790979138.525295 mapread 0x838080 thru 0x8386ec (0x66d bytes) 000073[0] 1790979138.565895 mapread 0x3d041e thru 0x3d0670 (0x253 bytes) 000074[0] 1790979138.573371 mapread 0x4e56a1 thru 0x4e581c (0x17c bytes) 000078[0] 1790979138.673738 punch from 0x56380b to 0x56495b (0x1151 bytes) truncating to largest ever: 0x80d9f5 000080[0] 1790979138.708973 trunc from 0x975275 to 0x80d9f4 (0x167880 bytes) 000082[0] 1790979138.785292 mapread 0x1ffa85 thru 0x201783 (0x1cff bytes) 000085[1] 1790979138.865681 read 0x04347f thru 0x044d57 (0x18d9 bytes) 000089[1] 1790979138.960551 read 0x4520c0 thru 0x4528b3 (0x7f4 bytes) 000091[0] 1790979138.996271 write 0x4e7994 thru 0x4e9651 (0x1cbe bytes) 000092[0] 1790979139.074290 read 0x3779c2 thru 0x3797eb (0x1e2a bytes) 000094[0] 1790979139.127429 write 0x1c1c63 thru 0x1c2890 (0xc2e bytes) 000095[1] 1790979139.155448 read 0x7ae5ad thru 0x7af4b0 (0xf04 bytes) 000096[1] 1790979139.195497 punch from 0x030e21 to 0x030e84 (0x64 bytes) 000097[0] 1790979139.236765 read 0x607bd0 thru 0x608648 (0xa79 bytes) 000099[1] 1790979139.254062 mapread 0x3ff224 thru 0x3ffc33 (0xa10 bytes) 000100[0] 1790979139.275356 punch from 0x00d0d9 to 0x00e036 (0xf5e bytes) 000102[0] 1790979139.303575 trunc from 0x80d9f5 to 0x1df10e (0x62e8e6 bytes) truncating to largest ever: 0x9745bb 000103[1] 1790979139.338592 trunc from 0x1df10f to 0x9745ba (0x7954ac bytes) 000104[0] 1790979139.417495 write 0x4995f8 thru 0x49a35c (0xd65 bytes) 000106[0] 1790979139.493425 mapread 0x773e02 thru 0x774735 (0x934 bytes) 000107[0] 1790979139.527241 write 0x900201 thru 0x900f4d (0xd4d bytes) 000108[1] 1790979139.570472 write 0x9827bd thru 0x984112 (0x1956 bytes) 000114[1] 1790979139.743470 write 0x673a4c thru 0x67439f (0x954 bytes) 000121[0] 1790979139.913337 read 0x164bcd thru 0x1651e6 (0x61a bytes) 000122[0] 1790979139.944477 punch from 0x189779 to 0x189d84 (0x60c bytes) 000124[1] 1790979139.989752 mapread 0x457563 thru 0x457b00 (0x59e bytes) 000125[1] 1790979140.038314 write 0x2f4f4d thru 0x2f5908 (0x9bc bytes) 000129[0] 1790979140.138935 mapwrite 0x1c73e2 thru 0x1c7779 (0x398 bytes) 000131[0] 1790979140.408056 mapwrite 0x6d8ea5 thru 0x6d9680 (0x7dc bytes) 000132[1] 1790979140.522932 read 0x1760c7 thru 0x1775d5 (0x150f bytes) 000133[1] 1790979140.550521 write 0x111182 thru 0x1111fb (0x7a bytes) 000134[0] 1790979140.580398 punch from 0x45abdd to 0x45b823 (0xc47 bytes) 000135[1] 1790979140.601219 write 0x513fa5 thru 0x5143ec (0x448 bytes) 000136[1] 1790979140.623219 write 0x7fc6bd thru 0x7fdcab (0x15ef bytes) 000137[1] 1790979140.652243 trunc from 0x984113 to 0x30f69e (0x674a74 bytes) 000140[1] 1790979140.727103 read 0x23e026 thru 0x23ef0c (0xee7 bytes) 000141[1] 1790979140.832554 read 0x2c2830 thru 0x2c360a (0xddb bytes) 000143[1] 1790979140.836351 mapread 0x1a54d6 thru 0x1a5c96 (0x7c1 bytes) 000144[0] 1790979140.838757 punch from 0x0379fd to 0x039018 (0x161c bytes) 000148[1] 1790979140.935263 mapread 0x069482 thru 0x06b276 (0x1df5 bytes) 000149[1] 1790979141.077685 read 0x1c9f0b thru 0x1cb188 (0x127e bytes) 000150[1] 1790979141.079731 read 0x24b258 thru 0x24bde2 (0xb8b bytes) 000151[1] 1790979141.080469 mapwrite 0x0ae0d4 thru 0x0afa64 (0x1991 bytes) 000157[0] 1790979141.438387 write 0x1cdcb0 thru 0x1cf38f (0x16e0 bytes) 000159[0] 1790979141.506529 mapwrite 0x11054f thru 0x11135f (0xe11 bytes) 000162[1] 1790979141.662311 trunc from 0x30f69f to 0x936223 (0x626b85 bytes) 000164[0] 1790979141.781918 mapwrite 0x801973 thru 0x802564 (0xbf2 bytes) 000166[1] 1790979141.998177 punch from 0x332483 to 0x333879 (0x13f7 bytes) 000167[1] 1790979142.043097 read 0x67eb2a thru 0x6805c8 (0x1a9f bytes) 000168[0] 1790979142.056943 mapread 0x0ca94c thru 0x0cb122 (0x7d7 bytes) 000170[1] 1790979142.128002 punch from 0x8f7aef to 0x8f7b2e (0x40 bytes) 000171[0] 1790979142.169288 write 0x252a49 thru 0x254115 (0x16cd bytes) 000172[0] 1790979142.234254 trunc from 0x936224 to 0x890fb3 (0xa5270 bytes) 000174[1] 1790979142.344552 write 0x2e3d9e thru 0x2e4e0d (0x1070 bytes) 000175[0] 1790979142.424435 punch from 0x08b874 to 0x08d51d (0x1caa bytes) 000176[0] 1790979142.488125 mapread 0x39b625 thru 0x39d5de (0x1fba bytes) 000177[0] 1790979142.511057 write 0x436a8a thru 0x437fed (0x1564 bytes) 000179[0] 1790979142.579642 write 0x38cda2 thru 0x38e883 (0x1ae2 bytes) 000180[0] 1790979142.688244 write 0x1caa65 thru 0x1cb581 (0xb1d bytes) 000181[1] 1790979142.708271 read 0x7dd221 thru 0x7dde53 (0xc33 bytes) 000182[1] 1790979142.759100 read 0x6d5e1b thru 0x6d6444 (0x62a bytes) 000184[0] 1790979142.841442 read 0x645571 thru 0x64560c (0x9c bytes) 000186[1] 1790979142.904642 punch from 0x2114d1 to 0x2132fc (0x1e2c bytes) 000188[1] 1790979143.023091 mapread 0x240148 thru 0x240429 (0x2e2 bytes) truncating to largest ever: 0x9a0dfd 000191[0] 1790979143.154696 trunc from 0x890fb4 to 0x9a0dfc (0x10fe49 bytes) 000192[0] 1790979143.266476 punch from 0x671041 to 0x672582 (0x1542 bytes) 000193[0] 1790979143.327361 mapwrite 0x721072 thru 0x7212e0 (0x26f bytes) 000194[1] 1790979143.444181 punch from 0x4895aa to 0x489ef7 (0x94e bytes) 000196[0] 1790979143.502553 mapwrite 0x39344c thru 0x394717 (0x12cc bytes) 000197[1] 1790979143.599350 trunc from 0x9a0dfd to 0x92652f (0x7a8cd bytes) 000200[1] 1790979143.793372 write 0x8de105 thru 0x8dfd63 (0x1c5f bytes) 000201[0] 1790979143.868492 mapread 0x1d349e thru 0x1d372b (0x28e bytes) 000202[0] 1790979143.971304 mapwrite 0x2e568c thru 0x2e621d (0xb92 bytes) 000203[1] 1790979144.155216 mapread 0x118a79 thru 0x119c34 (0x11bc bytes) 000204[0] 1790979144.245477 mapwrite 0x74fdae thru 0x7503f7 (0x64a bytes) 000205[1] 1790979144.363746 write 0x298e8e thru 0x29a304 (0x1477 bytes) 000206[1] 1790979144.421071 mapread 0x722733 thru 0x722d09 (0x5d7 bytes) 000207[0] 1790979144.465274 mapread 0x700e54 thru 0x702d63 (0x1f10 bytes) 000209[0] 1790979144.498375 mapread 0x2aecfe thru 0x2af11b (0x41e bytes) 000210[1] 1790979144.543279 trunc from 0x926530 to 0x1f24db (0x734054 bytes) truncating to largest ever: 0x9c8f1b 000211[0] 1790979144.605172 trunc from 0x1f24dc to 0x9c8f1a (0x7d6a3f bytes) 000212[1] 1790979144.654712 trunc from 0x9c8f1b to 0x00d26d (0x9bbcad bytes) 000214[1] 1790979144.756988 trunc from 0x00d26e to 0x5b3980 (0x5a6713 bytes) 000215[0] 1790979144.781400 read 0x4481a7 thru 0x449467 (0x12c1 bytes) 000216[1] 1790979144.833116 mapread 0x22ffbb thru 0x230f85 (0xfcb bytes) 000217[1] 1790979144.868541 read 0x0fd2a3 thru 0x0feec3 (0x1c21 bytes) 000218[1] 1790979144.889714 write 0x9eaeb6 thru 0x9ebcd5 (0xe20 bytes) 000222[1] 1790979144.946805 mapwrite 0x8b2d34 thru 0x8b41ae (0x147b bytes) 000223[0] 1790979145.158089 trunc from 0x9ebcd6 to 0x55ae5a (0x490e7b bytes) 000224[1] 1790979145.210269 mapwrite 0x811446 thru 0x8123bd (0xf78 bytes) 000226[1] 1790979145.366444 punch from 0x26ec80 to 0x2707e2 (0x1b63 bytes) 000227[1] 1790979145.371760 write 0x978987 thru 0x97a4e7 (0x1b61 bytes) 000228[1] 1790979145.403940 write 0x200ab3 thru 0x20288c (0x1dda bytes) 000231[0] 1790979145.465622 mapwrite 0x7b0626 thru 0x7b0776 (0x151 bytes) 000233[0] 1790979145.635063 read 0x2f6cee thru 0x2f7cae (0xfc1 bytes) 000234[1] 1790979145.674405 trunc from 0x97a4e8 to 0x49c50f (0x4ddfd8 bytes) 000238[1] 1790979145.837731 mapread 0x31d40d thru 0x31f019 (0x1c0d bytes) 000239[0] 1790979145.899356 punch from 0x19380a to 0x19444b (0xc42 bytes) 000240[0] 1790979145.939247 mapwrite 0x6693d5 thru 0x66aa5c (0x1688 bytes) 000241[1] 1790979146.042836 mapwrite 0x15ea40 thru 0x15f5a1 (0xb62 bytes) 000242[1] 1790979146.154317 mapread 0x0ca6dd thru 0x0cab50 (0x474 bytes) 000243[0] 1790979146.189464 mapwrite 0x15e7ec thru 0x15fcfa (0x150f bytes) 000246[0] 1790979146.292708 mapread 0x194fd1 thru 0x1959ec (0xa1c bytes) 000247[0] 1790979146.304108 punch from 0x316c69 to 0x31846f (0x1807 bytes) 000248[0] 1790979146.354216 trunc from 0x66aa5d to 0x6d301e (0x685c2 bytes) 000249[1] 1790979146.412136 mapread 0x607bb7 thru 0x608eb7 (0x1301 bytes) 000250[1] 1790979146.490199 trunc from 0x6d301f to 0x031cce (0x6a1350 bytes) 000251[0] 1790979146.597277 write 0x979c8a thru 0x97b449 (0x17c0 bytes) 000254[1] 1790979146.684337 write 0x5e82e0 thru 0x5e971e (0x143f bytes) 000255[0] 1790979146.714524 punch from 0x404845 to 0x405a70 (0x122c bytes) 000256[1] 1790979146.780624 read 0x7a78bd thru 0x7a9562 (0x1ca6 bytes) 000257[1] 1790979146.838738 mapwrite 0x58a454 thru 0x58b8af (0x145c bytes) 000259[1] 1790979147.163966 mapwrite 0x270ca4 thru 0x271a57 (0xdb4 bytes) 000260[1] 1790979147.324582 punch from 0x311a85 to 0x312cb7 (0x1233 bytes) 000262[0] 1790979147.337512 mapwrite 0x57d147 thru 0x57d90f (0x7c9 bytes) 000264[1] 1790979147.500386 write 0x870fd1 thru 0x8726a3 (0x16d3 bytes) 000265[0] 1790979147.548746 mapread 0x24521d thru 0x245915 (0x6f9 bytes) 000266[1] 1790979147.612369 read 0x083036 thru 0x083c4e (0xc19 bytes) 000269[1] 1790979147.657566 trunc from 0x97b44a to 0x8b3bd1 (0xc7878 bytes) 000270[0] 1790979147.720181 read 0x352819 thru 0x35364a (0xe32 bytes) 000273[1] 1790979147.763112 mapread 0x200e31 thru 0x2016df (0x8af bytes) 000275[0] 1790979147.813283 write 0x997c5c thru 0x9996fc (0x1aa1 bytes) 000277[1] 1790979147.871214 mapread 0x2c98bd thru 0x2cb80c (0x1f50 bytes) 000278[0] 1790979147.937458 punch from 0x1362b4 to 0x136c4d (0x99a bytes) 000280[1] 1790979148.002908 read 0x51886d thru 0x518aee (0x282 bytes) 000282[1] 1790979148.040555 mapread 0x4239dc thru 0x423a43 (0x68 bytes) truncating to largest ever: 0x9cbbdb 000283[1] 1790979148.092466 trunc from 0x9996fd to 0x9cbbda (0x324de bytes) 000286[1] 1790979148.132074 mapwrite 0x5c9494 thru 0x5cb015 (0x1b82 bytes) 000289[1] 1790979148.222585 write 0x34edc8 thru 0x34faf4 (0xd2d bytes) 000290[0] 1790979148.253113 mapread 0x60fb2b thru 0x6115f8 (0x1ace bytes) 000291[0] 1790979148.309101 write 0x5771e3 thru 0x578897 (0x16b5 bytes) 000292[0] 1790979148.360084 trunc from 0x9cbbdb to 0x0849e6 (0x9471f4 bytes) 000294[0] 1790979148.437585 read 0x03d805 thru 0x03ec52 (0x144e bytes) 000296[0] 1790979148.460212 punch from 0x043999 to 0x04440a (0xa72 bytes) 000300[1] 1790979148.561421 trunc from 0x0849e7 to 0x3991f9 (0x314813 bytes) 000302[1] 1790979148.639315 mapread 0x257ee3 thru 0x25845c (0x57a bytes) 000306[0] 1790979148.723933 punch from 0x1406f3 to 0x140a9d (0x3ab bytes) 000307[1] 1790979148.756404 mapread 0x0a5af8 thru 0x0a7a24 (0x1f2d bytes) 000308[1] 1790979148.829967 mapread 0x37b4b7 thru 0x37bd14 (0x85e bytes) 000313[1] 1790979148.906048 punch from 0x191b45 to 0x1924ee (0x9aa bytes) 000314[1] 1790979148.949113 mapread 0x393ad0 thru 0x394052 (0x583 bytes) 000320[1] 1790979149.146891 write 0x337005 thru 0x338698 (0x1694 bytes) 000321[0] 1790979149.199325 mapread 0x1e7aac thru 0x1e81af (0x704 bytes) 000322[0] 1790979149.380249 punch from 0x29a276 to 0x29bf32 (0x1cbd bytes) 000323[1] 1790979149.452744 mapread 0x013886 thru 0x013da0 (0x51b bytes) 000324[1] 1790979149.523502 mapwrite 0x4c36e9 thru 0x4c432b (0xc43 bytes) 000325[1] 1790979149.765269 trunc from 0x4c432c to 0x39d2f9 (0x127032 bytes) 000326[1] 1790979149.816742 mapwrite 0x070758 thru 0x072152 (0x19fb bytes) 000327[1] 1790979149.949162 read 0x0541d5 thru 0x056156 (0x1f82 bytes) 000328[1] 1790979150.041395 read 0x267b63 thru 0x268cb3 (0x1151 bytes) 000329[1] 1790979150.179966 write 0x9458d3 thru 0x945d26 (0x454 bytes) 000334[0] 1790979150.359477 trunc from 0x945d27 to 0x7cdc51 (0x1780d5 bytes) 000335[0] 1790979150.475219 punch from 0x7619eb to 0x761bcb (0x1e1 bytes) 000336[0] 1790979150.505511 write 0x4c8883 thru 0x4c94b4 (0xc32 bytes) 000337[1] 1790979150.714465 trunc from 0x7cdc52 to 0x91d086 (0x14f435 bytes) 000338[0] 1790979150.784472 read 0x5ca294 thru 0x5cbdaf (0x1b1c bytes) 000339[1] 1790979150.812771 mapwrite 0x2f3b81 thru 0x2f3f99 (0x419 bytes) 000341[0] 1790979151.032614 read 0x86148d thru 0x8615c8 (0x13c bytes) 000344[0] 1790979151.073366 read 0x6b963a thru 0x6ba2fc (0xcc3 bytes) 000347[0] 1790979151.085250 punch from 0x1eb089 to 0x1eb892 (0x80a bytes) 000348[0] 1790979151.119165 read 0x121783 thru 0x121dc8 (0x646 bytes) 000350[0] 1790979151.127181 mapwrite 0x0ea510 thru 0x0ebd6f (0x1860 bytes) 000351[0] 1790979151.224696 read 0x36a44b thru 0x36b7e1 (0x1397 bytes) 000354[0] 1790979151.235780 punch from 0x05c43d to 0x05d784 (0x1348 bytes) 000356[1] 1790979151.289759 punch from 0x1fc23b to 0x1fd0ec (0xeb2 bytes) 000357[0] 1790979151.318252 mapwrite 0x560e74 thru 0x5614ad (0x63a bytes) 000360[1] 1790979151.626003 read 0x6d6ef0 thru 0x6d785d (0x96e bytes) 000362[1] 1790979151.672820 trunc from 0x91d087 to 0x394bb6 (0x5884d0 bytes) 000364[1] 1790979151.824527 trunc from 0x394bb7 to 0x24059a (0x15461c bytes) 000366[0] 1790979151.885240 trunc from 0x24059b to 0x54ec02 (0x30e668 bytes) 000367[1] 1790979151.950527 punch from 0x2d1079 to 0x2d2e0a (0x1d92 bytes) 000372[0] 1790979152.062444 trunc from 0x54ec03 to 0x6ce163 (0x17f561 bytes) 000373[1] 1790979152.134677 mapwrite 0x8ceff1 thru 0x8cfc1b (0xc2b bytes) 000375[0] 1790979152.270116 read 0x56810e thru 0x569518 (0x140b bytes) 000376[0] 1790979152.296690 punch from 0x05c2fc to 0x05dab3 (0x17b8 bytes) 000377[0] 1790979152.333653 punch from 0x2789f6 to 0x278e05 (0x410 bytes) 000381[0] 1790979152.377186 punch from 0x11d4a7 to 0x11e3e4 (0xf3e bytes) 000383[0] 1790979152.386158 mapwrite 0x3a46c7 thru 0x3a5729 (0x1063 bytes) 000385[1] 1790979152.453119 punch from 0x274707 to 0x275ba0 (0x149a bytes) 000386[0] 1790979152.481246 punch from 0x395e80 to 0x39773d (0x18be bytes) 000388[0] 1790979152.516446 punch from 0x7d0483 to 0x7d1df9 (0x1977 bytes) 000390[1] 1790979152.558030 read 0x314173 thru 0x3149ad (0x83b bytes) 000391[0] 1790979152.591117 mapwrite 0x1f8a5b thru 0x1fa1c8 (0x176e bytes) 000392[0] 1790979152.690068 mapread 0x676266 thru 0x677a7f (0x181a bytes) 000394[0] 1790979152.742417 read 0x046b44 thru 0x047ba0 (0x105d bytes) 000396[1] 1790979152.804331 trunc from 0x8cfc1c to 0x08e15d (0x841abe bytes) 000397[0] 1790979152.898385 mapwrite 0x993ada thru 0x994032 (0x559 bytes) 000399[0] 1790979153.096453 punch from 0x7210e1 to 0x721480 (0x3a0 bytes) 000401[0] 1790979153.158022 read 0x8f19b4 thru 0x8f1cec (0x339 bytes) 000402[0] 1790979153.189340 trunc from 0x994033 to 0x299d1e (0x6fa314 bytes) 000405[0] 1790979153.219108 trunc from 0x299d1f to 0x4d374a (0x239a2c bytes) 000406[0] 1790979153.248114 mapwrite 0x605aa6 thru 0x6071c7 (0x1722 bytes) 000407[0] 1790979153.470456 punch from 0x220190 to 0x221be9 (0x1a5a bytes) 000408[1] 1790979153.494481 mapread 0x20c592 thru 0x20dd9a (0x1809 bytes) 000410[0] 1790979153.567472 mapread 0x5b8267 thru 0x5b8bec (0x986 bytes) 000413[0] 1790979153.589188 read 0x199e59 thru 0x19ba5f (0x1c07 bytes) 000415[1] 1790979153.606203 read 0x3da00c thru 0x3db65c (0x1651 bytes) 000416[1] 1790979153.679454 mapwrite 0x9f472f thru 0x9f61f6 (0x1ac8 bytes) 000417[0] 1790979153.992105 punch from 0x5cd169 to 0x5cdb42 (0x9da bytes) 000419[0] 1790979154.164592 mapread 0x58a613 thru 0x58c271 (0x1c5f bytes) 000422[1] 1790979154.311475 trunc from 0x9f61f7 to 0x32a332 (0x6cbec4 bytes) 000423[0] 1790979154.346701 write 0x39850a thru 0x3998eb (0x13e2 bytes) 000428[0] 1790979154.477451 read 0x229cfb thru 0x229e0b (0x111 bytes) 000430[1] 1790979154.669640 trunc from 0x3998ec to 0x1dfddb (0x1b9b10 bytes) 000431[1] 1790979154.737092 read 0x1bcc3a thru 0x1bdb44 (0xf0b bytes) 000432[0] 1790979154.816262 read 0x0ee5f8 thru 0x0efa0e (0x1417 bytes) 000434[1] 1790979154.947124 read 0x1c1e7a thru 0x1c2942 (0xac9 bytes) 000435[0] 1790979155.059439 mapread 0x1d1599 thru 0x1d2220 (0xc88 bytes) 000436[1] 1790979155.113610 read 0x16c98b thru 0x16e587 (0x1bfd bytes) 000438[1] 1790979155.141330 punch from 0x1c4373 to 0x1c52b1 (0xf3f bytes) 000443[1] 1790979155.265951 mapread 0x1b2e72 thru 0x1b35fb (0x78a bytes) 000446[1] 1790979155.354466 trunc from 0x1dfddc to 0x34d1b6 (0x16d3db bytes) 000449[1] 1790979155.410108 mapread 0x26e0fe thru 0x26fb5f (0x1a62 bytes) 000450[1] 1790979155.454200 read 0x25450e thru 0x254b6c (0x65f bytes) 000451[1] 1790979155.478267 punch from 0x0e1e6e to 0x0e3b09 (0x1c9c bytes) 000452[1] 1790979155.484737 mapread 0x223fc5 thru 0x2253a4 (0x13e0 bytes) 000457[1] 1790979155.561075 trunc from 0x34d1b7 to 0x1f8e3f (0x154377 bytes) 000459[0] 1790979155.616420 trunc from 0x1f8e40 to 0x0dc742 (0x11c6fd bytes) 000460[0] 1790979155.675501 mapwrite 0x80448a thru 0x804a10 (0x587 bytes) 000461[0] 1790979155.873896 punch from 0x0e3b6e to 0x0e5378 (0x180b bytes) 000464[0] 1790979155.980304 punch from 0x3b306b to 0x3b3405 (0x39b bytes) 000468[0] 1790979156.081700 mapread 0x18da4a thru 0x18f5a6 (0x1b5d bytes) 000469[1] 1790979156.131251 trunc from 0x804a11 to 0x7cc78c (0x38284 bytes) 000471[0] 1790979156.239934 read 0x2d2985 thru 0x2d3c00 (0x127c bytes) 000472[1] 1790979156.264196 punch from 0x215cf5 to 0x21780b (0x1b17 bytes) 000475[0] 1790979156.379439 trunc from 0x7cc78d to 0x815722 (0x48f96 bytes) 000476[0] 1790979156.457358 mapwrite 0x0f5ad3 thru 0x0f646b (0x999 bytes) 000477[0] 1790979156.539218 write 0x8b79aa thru 0x8b87cc (0xe23 bytes) 000478[1] 1790979156.546761 read 0x7a172a thru 0x7a2b15 (0x13ec bytes) 000479[0] 1790979156.578504 write 0x598af7 thru 0x59a42d (0x1937 bytes) 000481[0] 1790979156.662066 punch from 0x1f330f to 0x1f44fc (0x11ee bytes) 000482[1] 1790979156.708228 mapwrite 0x3e668e thru 0x3e6ccb (0x63e bytes) 000483[0] 1790979156.835348 punch from 0x179a92 to 0x17b910 (0x1e7f bytes) 000484[0] 1790979156.876757 trunc from 0x8b87cd to 0x2b2489 (0x606343 bytes) 000486[0] 1790979156.977694 write 0x609fd5 thru 0x60bc48 (0x1c74 bytes) 000487[0] 1790979157.020683 write 0x39c9e4 thru 0x39d30f (0x92c bytes) 000488[1] 1790979157.064391 trunc from 0x60bc49 to 0x0d94d9 (0x53276f bytes) 000489[1] 1790979157.202178 trunc from 0x0d94da to 0x8d5fec (0x7fcb13 bytes) 000490[0] 1790979157.259272 write 0x661e5f thru 0x662e34 (0xfd6 bytes) 000492[1] 1790979157.320612 trunc from 0x8d5fed to 0x4cc490 (0x409b5c bytes) 000493[0] 1790979157.393483 read 0x4175ee thru 0x418b1d (0x1530 bytes) 000494[0] 1790979157.417173 read 0x38bd62 thru 0x38dc00 (0x1e9f bytes) 000495[1] 1790979157.442326 read 0x3703a0 thru 0x37217d (0x1dde bytes) 000497[0] 1790979157.449145 punch from 0x222224 to 0x2228a7 (0x684 bytes) 000498[0] 1790979157.500284 mapread 0x193cc5 thru 0x194620 (0x95c bytes) 000499[1] 1790979157.584783 write 0x516c7e thru 0x51828a (0x160d bytes) All operations completed A-OK! PASS 16b (35s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 18:12:44 (1790979164) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 18:12:49 (1790979169) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.42339 s, 19.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.397484 s, 21.1 MB/s ldlm.namespaces.MGC192.168.203.105@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9e3d8586d800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9e3d91ae8800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9e3d91ae8800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9e3d91ae8800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.7311 s, 7.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.97666 s, 26.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.03931 s, 34.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.50316 s, 14.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6759 s, 9.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.28407 s, 24.5 MB/s PASS 16d (59s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 18:13:47 (1790979227) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.31926 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237887 s, 4.4 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 18:13:56 (1790979236) TIMEOUT PASS 16f (29s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 18:14:24 (1790979264) TIMEOUT PASS 16g (30s) == sanityn test 16h: mmap read after truncate file ======= 18:14:54 (1790979294) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (8s) == sanityn test 16i: read after truncate file ============ 18:15:02 (1790979302) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (9s) == sanityn test 16j: race dio with buffered i/o ========== 18:15:11 (1790979311) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.4806 s, 17.7 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.57666 s, 16.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.06572 s, 8.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.41269 s, 10.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.0694 s, 8.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.72891 s, 9.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.79427 s, 6.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.38159 s, 11.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.67468 s, 9.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.10436 s, 6.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.16136 s, 6.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.20594 s, 8.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.4404 s, 7.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.79591 s, 6.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.29043 s, 6.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.01531 s, 6.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.53019 s, 5.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.52319 s, 5.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.56069 s, 5.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (46s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 18:15:57 (1790979357) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Fri Oct 2 18:16:59 EDT 2026 All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 2 done: rc=0 Fri Oct 2 18:17:00 EDT 2026 + fsx 3 done: rc=0 Fri Oct 2 18:17:00 EDT 2026 + fsx 4 done: rc=0 Fri Oct 2 18:17:00 EDT 2026 + fsx 5 done: rc=0 Fri Oct 2 18:17:00 EDT 2026 PASS 16k (74s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 18:17:11 (1790979431) fail_loc=0x8000030a PASS 17 (12s) == sanityn test 18: mmap sanity check =========================================================================================== 18:17:24 (1790979444) mmap test1: basic mmap operation (PASS, 0.25133s) mmap test2: MAP_PRIVATE not write back (PASS, 0.17218s) mmap test3: concurrent mmap ops on two nodes (PASS, 6.2744s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.6538s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.30328s) mmap test6: check mmap write/read content on two nodes (PASS, 0.36591s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.9324s) mmap test11: multi-thread shared mmap (PASS, 2.2852s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1527s) PASS 18 (27s) == sanityn test 19: test concurrent uncached read races ========================================================================= 18:17:52 (1790979472) oleg305-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (6s) == sanityn test 20: test extra readahead page left in cache ============================================================== 18:17:57 (1790979477) PASS 20 (8s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 18:18:05 (1790979485) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (7s) == sanityn test 23: others should see updated atime while another read============================================================== 18:18:13 (1790979493) now is 1790979496 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7617 new atime is 1790979557 PASS 23 (72s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 18:19:24 (1790979564) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 386 524.5K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 386 524.5K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.0M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537447 386 537061 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 386 524.5K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537447 386 537061 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537447 386 537061 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 18:19:32 (1790979572) PASS 24b (8s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 18:19:40 (1790979580) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (8s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 18:19:48 (1790979588) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 18:19:51 (1790979591) utime: good mknod times 1790979593 <= 1790979594 <= 1790979594 for /mnt/lustre/f26a utime: good mknod times 1790979593 <= 1790979594 <= 1790979594 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (9s) == sanityn test 26b: sync mtime between ost and mds ====== 18:20:01 (1790979601) PASS 26b (10s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 18:20:11 (1790979611) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7617 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-10-02 18:20:13.000000000 -0400 Birth: 2026-10-02 18:20:14.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-10-02 18:20:13.000000000 -0400 Birth: 2026-10-02 18:20:14.000000000 -0400 PASS 26c (9s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 18:20:22 (1790979622) 0 PASS 30 (10s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 18:20:32 (1790979632) fail_loc=0x314 PASS 31a (9s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 18:20:41 (1790979641) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg305-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.105@tcp:/lustre 7542784 40960 7497728 1% /mnt/lustre2 PASS 31b (25s) == sanityn test 31r: open-rename(replace) race =========== 18:21:06 (1790979666) -rw-r--r-- 1 root root 0 Oct 2 18:21 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (11s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 18:21:17 (1790979677) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1211: 43202 Terminated tail -f $tfile PASS 31s (8s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 18:21:25 (1790979685) PASS 31t (9s) == sanityn test 32b: lockless i/o ======================== 18:21:34 (1790979694) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (5s) == sanityn test 32c: contention detection ================ 18:21:39 (1790979699) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.10968 s, 24.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.60604 s, 32.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.50539 s, 20.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.98316 s, 17.6 MB/s contention_seconds 4, contention_events 17 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.09577 s, 16.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.33541 s, 15.7 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.81878 s, 28.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.86227 s, 28.2 MB/s PASS 32c (37s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 18:22:18 (1790979738) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 18:22:22 (1790979742) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 18:22:26 (1790979746) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 18:22:30 (1790979750) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (4s) == sanityn test 34: no lock timeout under IO ============= 18:22:34 (1790979754) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg305-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e3d8586d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e3d8586d800.ost_server_uuid in IDLE state after 0 sec oleg305-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e3d8586d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e3d8586d800.ost_server_uuid in IDLE state after 0 sec oleg305-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e3d8586d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e3d8586d800.ost_server_uuid in IDLE state after 0 sec oleg305-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e3d8586d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e3d8586d800.ost_server_uuid in IDLE state after 0 sec oleg305-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e3d8586d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e3d8586d800.ost_server_uuid in IDLE state after 0 sec oleg305-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e3d8586d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e3d8586d800.ost_server_uuid in IDLE state after 0 sec PASS 34 (145s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 18:24:59 (1790979899) fail_loc=0x80000317 Race attempt 0 Wait for 50518 50558 for 60 sec... - open/close 677 (time 1790979917.34 total 10.01 last 67.62) - open/close 1367 (time 1790979927.35 total 20.02 last 68.94) - open/close 2027 (time 1790979937.35 total 30.03 last 65.97) - open/close 2717 (time 1790979947.36 total 40.04 last 68.92) - open/close 3375 (time 1790979957.37 total 50.04 last 65.76) total: 4000 open/close in 59.51 seconds: 67.22 ops/second /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1861: 50518 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.105@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.203.105@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (74s) == sanityn test 36: handle ESTALE/open-unlink correctly == 18:26:13 (1790979973) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg305-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.59922 s, 32.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7617 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.4494 s, 36.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7617 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7493632) after_dd(7444480) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.34942 s, 38.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7617 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7497728) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.59073 s, 33.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7617 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7493632) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.06905 s, 25.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7617 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.63883 s, 32.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7617 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7497728) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.81098 s, 29.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7617 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7493632) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.6083 s, 32.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7617 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7491584) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.24553 s, 23.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7617 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(8) *** before(7493632) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.90444 s, 27.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7617 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7495680) after_dd(7442432) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.52404 s, 34.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7617 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7497728) after_dd(7439360) after(7490560) PASS 36 (346s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 18:31:59 (1790980319) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7617 - create 1202 (time 1790980334.05 total 10.01 last 120.14) - create 2180 (time 1790980344.06 total 20.01 last 97.71) - create 3326 (time 1790980354.07 total 30.02 last 114.54) - create 4550 (time 1790980364.08 total 40.03 last 122.29) - create 5766 (time 1790980374.08 total 50.03 last 121.55) - create 6894 (time 1790980384.09 total 60.04 last 112.72) - create 8131 (time 1790980394.09 total 70.05 last 123.62) - create 9315 (time 1790980404.11 total 80.06 last 118.24) total: 10000 create in 85.32 seconds: 117.20 ops/second PASS 37 (98s) == sanityn test 39a: file mtime does not change after rename ========================================================== 18:33:37 (1790980417) repeat after cancel_lru_locks PASS 39a (9s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 18:33:47 (1790980427) repeat after cancel_lru_locks PASS 39b (10s) == sanityn test 39c: check truncate mtime update ================================================================================ 18:33:57 (1790980437) repeat after cancel_lru_locks PASS 39c (9s) == sanityn test 39d: sync write should update mtime ====== 18:34:05 (1790980445) fail_loc=0x411 fail_loc=0 PASS 39d (8s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 18:34:15 (1790980455) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (19s) == sanityn test 40c: pdirops: link and others ======================================================================== 18:34:34 (1790980474) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (18s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 18:34:53 (1790980493) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (18s) == sanityn test 40e: pdirops: rename and others ======================================================================== 18:35:12 (1790980512) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (19s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 18:35:31 (1790980531) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (14s) == sanityn test 41b: pdirops: create vs create ======================================================================== 18:35:45 (1790980545) open: File exists Conflict PASS 41b (14s) == sanityn test 41c: pdirops: create vs link ======================================================================== 18:35:59 (1790980559) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 18:36:13 (1790980573) Conflict PASS 41d (13s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 18:36:27 (1790980587) Conflict PASS 41e (15s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 18:36:43 (1790980603) Conflict PASS 41f (15s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 18:36:57 (1790980617) Conflict PASS 41g (13s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 18:37:10 (1790980630) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2433: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (17s) == sanityn test 41i: reint_open: create vs create ======== 18:37:27 (1790980647) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1124s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 18:56:11 (1790981771) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (14s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 18:56:25 (1790981785) open: File exists Conflict PASS 42b (15s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 18:56:40 (1790981800) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (17s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 18:56:57 (1790981817) Conflict PASS 42d (17s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 18:57:14 (1790981834) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (15s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 18:57:29 (1790981849) Conflict PASS 42f (15s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 18:57:44 (1790981864) Conflict PASS 42g (16s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 18:58:00 (1790981880) Conflict PASS 42h (14s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 18:58:14 (1790981894) PASS 43a (83s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 18:59:37 (1790981977) Conflict PASS 43b (15s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 18:59:52 (1790981992) Conflict PASS 43c (15s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 19:00:07 (1790982007) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (14s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 19:00:21 (1790982021) Conflict PASS 43e (15s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 19:00:36 (1790982036) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (14s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 19:00:50 (1790982050) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (15s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 19:01:05 (1790982065) Conflict PASS 43h (15s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 19:01:20 (1790982080) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 19:01:24 (1790982084) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (137s) == sanityn test 43k: unlink vs create ==================== 19:03:42 (1790982222) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1243s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 19:24:24 (1790983464) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Oct 2 19:24:32 EDT 2026 PASS 44a (13s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 19:24:37 (1790983477) open: File exists Conflict PASS 44b (13s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 19:24:50 (1790983490) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (12s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 19:25:03 (1790983503) Conflict PASS 44d (13s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 19:25:16 (1790983516) Conflict PASS 44e (14s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 19:25:30 (1790983530) Conflict PASS 44f (14s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 19:25:44 (1790983544) Conflict PASS 44g (14s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 19:25:58 (1790983558) Conflict PASS 44h (13s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 19:26:11 (1790983571) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (3s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 19:26:14 (1790983574) PASS 45a (140s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 19:28:34 (1790983714) Conflict PASS 45b (15s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 19:28:49 (1790983729) Conflict PASS 45c (14s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 19:29:03 (1790983743) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (14s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 19:29:18 (1790983758) Conflict PASS 45e (16s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 19:29:34 (1790983774) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (15s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 19:29:49 (1790983789) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (14s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 19:30:03 (1790983803) Conflict PASS 45h (13s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 19:30:16 (1790983816) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 19:30:20 (1790983820) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1249s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 19:51:10 (1790985070) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (14s) == sanityn test 46b: pdirops: link vs create ======================================================================== 19:51:23 (1790985083) open: File exists Conflict PASS 46b (13s) == sanityn test 46c: pdirops: link vs link ======================================================================== 19:51:36 (1790985096) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (13s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 19:51:49 (1790985109) Conflict PASS 46d (13s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 19:52:02 (1790985122) Conflict PASS 46e (13s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 19:52:15 (1790985135) Conflict PASS 46f (14s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 19:52:29 (1790985149) Conflict PASS 46g (14s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 19:52:43 (1790985163) Conflict PASS 46h (14s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 19:52:58 (1790985178) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (4s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 19:53:01 (1790985181) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (3s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 19:53:04 (1790985184) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 19:53:08 (1790985188) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 19:53:11 (1790985191) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 19:53:14 (1790985194) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (4s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 19:53:19 (1790985199) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (4s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 19:53:22 (1790985202) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (4s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 19:53:26 (1790985206) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.120967 s, 84.7 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (11s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 19:53:38 (1790985218) 0+1 records in 0+1 records out 158 bytes copied, 0.0357455 s, 4.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3696: kill: (218847) - No such process PASS 51a (8s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 19:53:46 (1790985226) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0164796 s, 62.1 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747961 s, 14.0 MB/s 3145728 PASS 51b (20s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 19:54:06 (1790985246) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.040973 s, 25.0 kB/s PASS 51c (13s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 19:54:19 (1790985259) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381692 s, 27.5 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (8s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 19:54:27 (1790985267) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 read lease(1) has applied. PASS 51e (10s) == sanityn test 54: rename locking ======================= 19:54:37 (1790985277) llite.lustre-ffff9e3d8586d800.xattr_cache=1 llite.lustre-ffff9e3d91ae8800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 14 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 19:54:39.000000000 -0400 Modify: 2026-10-02 19:54:39.000000000 -0400 Change: 2026-10-02 19:54:39.000000000 -0400 Birth: 2026-10-02 19:54:40.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 14 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 19:54:45.000000000 -0400 Modify: 2026-10-02 19:54:46.000000000 -0400 Change: 2026-10-02 19:54:46.000000000 -0400 Birth: 2026-10-02 19:54:46.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 2 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 19:54:52.000000000 -0400 Modify: 2026-10-02 19:54:52.000000000 -0400 Change: 2026-10-02 19:54:52.000000000 -0400 Birth: 2026-10-02 19:54:52.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 14 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 19:54:52.000000000 -0400 Modify: 2026-10-02 19:54:53.000000000 -0400 Change: 2026-10-02 19:54:53.000000000 -0400 Birth: 2026-10-02 19:54:52.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 14 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 19:54:58.000000000 -0400 Modify: 2026-10-02 19:54:59.000000000 -0400 Change: 2026-10-02 19:54:59.000000000 -0400 Birth: 2026-10-02 19:54:59.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 14 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 19:54:58.000000000 -0400 Modify: 2026-10-02 19:54:59.000000000 -0400 Change: 2026-10-02 19:54:59.000000000 -0400 Birth: 2026-10-02 19:54:59.000000000 -0400 PASS 54 (34s) == sanityn test 55a: rename vs unlink target dir ========= 19:55:11 (1790985311) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (14s) == sanityn test 55b: rename vs unlink source dir ========= 19:55:25 (1790985325) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (14s) == sanityn test 55c: rename vs unlink orphan target dir == 19:55:39 (1790985339) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (20s) == sanityn test 55d: rename file vs link ================= 19:55:59 (1790985359) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (16s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 19:56:15 (1790985375) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (5s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 19:56:20 (1790985380) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (19s) == sanityn test 55g: rename: race with trylock =========== 19:56:39 (1790985399) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (26s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 19:57:05 (1790985425) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1790985428 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.540169 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (123s) SKIP: sanityn test_56b skipping ALWAYS excluded test 56b == sanityn test 60: Verify data_version behaviour ======== 19:59:10 (1790985550) PASS 60 (8s) == sanityn test 70a: cd directory && rm directory ======== 19:59:18 (1790985558) PASS 70a (7s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 19:59:25 (1790985565) PASS 70b (8s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 19:59:33 (1790985573) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00718046 s, 9.1 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 19:59:36 (1790985576) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0289223 s, 2.3 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 19:59:40 (1790985580) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (4s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 19:59:43 (1790985583) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0166339 s, 3.9 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (2s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 19:59:46 (1790985586) llite.lustre-ffff9e3d8586d800.xattr_cache=1 llite.lustre-ffff9e3d91ae8800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (8s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 19:59:54 (1790985594) llite.lustre-ffff9e3d8586d800.xattr_cache=1 llite.lustre-ffff9e3d91ae8800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (7s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 20:00:01 (1790985601) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0199136 s, 51.4 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0294175 s, 34.8 kB/s 231204: get lock2 231205: get lock1 231204: done 231204: get lock1 231205: done 231205: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231205: cannot get lock2: Resource deadlock avoided 231205: exit rc=0 231204: done 231204: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 20:00:12 (1790985612) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.101386 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385949 s, 27.2 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4379: 231785 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (12s) == sanityn test 76: Verify MDT open_files listing ======== 20:00:24 (1790985624) opening files: ........ closing files: ........ PASS 76 (80s) == sanityn test 77a: check FIFO NRS policy =============== 20:01:44 (1790985704) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.486878 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615296 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138229 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111111 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164153 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950422 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145228 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200357 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197504 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149584 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224363 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253918 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241602 s, 4.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326095 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315806 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277582 s, 37.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310403 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411617 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101269 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230866 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057783 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124535 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918207 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147664 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951092 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951713 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102049 s, 10.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135131 s, 7.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00679129 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110937 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127645 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.476004 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.551997 s, 1.9 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (9s) == sanityn test 77b: check CRR-N NRS policy ============== 20:01:53 (1790985713) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.411775 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568374 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959585 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10721 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123268 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680966 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217454 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312587 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266301 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328014 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271693 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340705 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392296 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31622 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353507 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35191 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368259 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195423 s, 53.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837271 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109137 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428811 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776562 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00652353 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507256 s, 20.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124382 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703324 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147366 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00914971 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00934809 s, 112 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0099549 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136813 s, 7.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101728 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102955 s, 10.2 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg305-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg305-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.621274 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116846 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732365 s, 14.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106203 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112657 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140821 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150901 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184622 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871117 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130195 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143459 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109293 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23019 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228974 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332697 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234089 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200201 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00371051 s, 283 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632029 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642439 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922084 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00344885 s, 304 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664577 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881117 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115652 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080589 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402125 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367321 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301397 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251315 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355163 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287863 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491115 s, 21.4 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg305-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg305-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (12s) == sanityn test 77c: check ORR NRS policy ================ 20:02:05 (1790985725) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.435491 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118332 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111383 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117377 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189847 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197013 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201035 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244515 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222981 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242212 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24984 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240568 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270415 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252675 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244569 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242802 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270143 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207102 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125861 s, 83.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715138 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110376 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0040838 s, 257 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160396 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807468 s, 13.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392893 s, 26.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410554 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015688 s, 66.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104022 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243926 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161149 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155225 s, 67.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341199 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00832452 s, 126 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg305-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg305-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.423147 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677272 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493025 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781271 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130494 s, 8.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16429 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186631 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159721 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250337 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282336 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250186 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274383 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221165 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281144 s, 3.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246899 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250969 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345736 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430868 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164483 s, 63.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958764 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235223 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301568 s, 34.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13096 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14258 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03762 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573276 s, 18.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470986 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593414 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499674 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508779 s, 20.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524076 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540759 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915016 s, 11.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.40675 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591354 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738661 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169616 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186187 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205346 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241885 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240197 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241193 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286676 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290737 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28459 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100392 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317372 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286263 s, 3.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307915 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952683 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463002 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458224 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987105 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121054 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828669 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568013 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574395 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697724 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652632 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573648 s, 18.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755681 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781441 s, 13.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263502 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35707 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00278298 s, 377 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351031 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=8 removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg305-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg305-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (14s) == sanityn test 77d: check TRR nrs policy ================ 20:02:19 (1790985739) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.490615 s, 34.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483484 s, 21.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101606 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119134 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129962 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838941 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122056 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133071 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156312 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150803 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164404 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179854 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25566 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263721 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229735 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252224 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366181 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113234 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556345 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00790329 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00624922 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929602 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00715802 s, 146 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843651 s, 12.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153916 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106357 s, 98.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106155 s, 98.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144729 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663551 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447668 s, 23.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511527 s, 20.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391867 s, 26.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742918 s, 14.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg305-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg305-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.52863 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732936 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104889 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138672 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240123 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238452 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241266 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211093 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207281 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282338 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241769 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338678 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32805 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260761 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329811 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392108 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234469 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00608576 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318903 s, 32.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357541 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.464813 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419408 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00535311 s, 196 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829922 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964685 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761121 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100194 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616344 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972399 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599976 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058675 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107326 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013622 s, 77.0 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.506444 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533126 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469532 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100096 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149932 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112703 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115694 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141323 s, 7.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134063 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242191 s, 4.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223872 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2058 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180479 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212191 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21612 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256322 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234479 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00302056 s, 347 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00301688 s, 348 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00645072 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154146 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11295 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137729 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1245 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977566 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112866 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996144 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164341 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837249 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063602 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642539 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536085 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534143 s, 19.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg305-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg305-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (15s) == sanityn test 77e: check TBF NID nrs policy ============ 20:02:34 (1790985754) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.5@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.320112 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068657 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779234 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908048 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714078 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135305 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213673 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192485 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141333 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225737 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279115 s, 3.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302099 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321807 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315152 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329823 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336338 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167869 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193861 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197675 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180174 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175189 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202253 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18027 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181264 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158128 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158785 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164266 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202609 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166895 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162439 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158502 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174258 s, 6.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.492376 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059208 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728815 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15839 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177742 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194365 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208603 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158246 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182426 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165618 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216075 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162274 s, 6.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204916 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245695 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209395 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233605 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221639 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217005 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216311 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199948 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222488 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210266 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194007 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195754 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227581 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214257 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223385 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211768 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207165 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215105 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210802 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218878 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268489 s, 3.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.545796 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787496 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848503 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871551 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718784 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160606 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169479 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200838 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216484 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145033 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244785 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169179 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290978 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29202 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197657 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255459 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250767 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699587 s, 15.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683636 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340112 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421017 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587758 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414249 s, 25.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689168 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463727 s, 22.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646307 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047802 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593602 s, 17.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392433 s, 26.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971619 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513836 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627214 s, 16.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.409039 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503471 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120452 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167435 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104454 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124622 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147735 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129313 s, 8.1 MB/sldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159513 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181339 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218351 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168894 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194649 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244142 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219378 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227521 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34348 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062666 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407336 s, 25.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471235 s, 22.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101751 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627182 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350323 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874438 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583157 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513342 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900995 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689084 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480158 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955254 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567837 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709349 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251304 s, 41.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (21s) == sanityn test 77f: check TBF JobID nrs policy ========== 20:02:55 (1790985775) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg305-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.476198 s, 35.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249617 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100846 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105723 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121813 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777047 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113693 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154155 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179645 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209418 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158209 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201037 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106599 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221655 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239989 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25256 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287893 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998397 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603539 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672111 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141291 s, 74.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647971 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327698 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075629 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639556 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120687 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677844 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130033 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281162 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478332 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101148 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716776 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127491 s, 82.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg305-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.573698 s, 29.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114568 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645082 s, 16.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274849 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404809 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355323 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347815 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354082 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406526 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412795 s, 2.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402806 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437781 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.501427 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.458537 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461311 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00556968 s, 188 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23687 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511128 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539721 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00718358 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.475006 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208468 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773635 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.432693 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00467198 s, 224 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00822783 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124705 s, 84.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00245609 s, 427 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.522765 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00778185 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0024997 s, 419 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045202 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526711 s, 19.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg305-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.466803 s, 35.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642333 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480371 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199612 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163963 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192836 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230823 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226489 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261897 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224359 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268367 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255409 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262593 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256301 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300859 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314737 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304526 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385181 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073276 s, 14.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755619 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00533256 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504043 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634499 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00237548 s, 441 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11873 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125309 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119626 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298177 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382724 s, 27.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269075 s, 39.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230235 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162955 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218495 s, 48.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg305-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg305-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg305-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.585166 s, 28.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968726 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109376 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929504 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28787 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2624 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270471 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234653 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322707 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292483 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316829 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344034 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292062 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341545 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294276 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32135 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421649 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625509 s, 16.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10459 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521689 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815412 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878897 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972224 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055788 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062393 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703008 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141304 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132623 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178453 s, 58.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962225 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101207 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158953 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109157 s, 9.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (25s) == sanityn test 77g: Change TBF type directly ============ 20:03:20 (1790985800) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (12s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 20:03:32 (1790985812) oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg305-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg305-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg305-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg305-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 PASS 77h (12s) == sanityn test 77i: Change rank of TBF rule ============= 20:03:44 (1790985824) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (26s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 20:04:11 (1790985851) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.499758 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540871 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137071 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154206 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13312 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149762 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121763 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133082 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158424 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232304 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22122 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053091 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827879 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214841 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27615 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251981 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816568 s, 12.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049992 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270841 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299201 s, 35.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305858 s, 34.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297202 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067226 s, 15.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725878 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00307055 s, 341 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00908867 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538181 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00505157 s, 208 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00349576 s, 300 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00546088 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165354 s, 63.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786505 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00497206 s, 211 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg305-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg305-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.25409 s, 11.3 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5762 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (67s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 20:05:18 (1790985918) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg305-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg305-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg305-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg305-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg305-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.25848 s, 13.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837871 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979888 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506282 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119885 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151938 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166546 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192533 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146337 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130039 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189991 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137783 s, 7.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146519 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258777 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209678 s, 5.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162124 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267464 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302916 s, 34.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710021 s, 14.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444308 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789335 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759531 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562878 s, 18.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811826 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679646 s, 15.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447738 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457328 s, 22.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585921 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780991 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846718 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801492 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030817 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901992 s, 11.6 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9543 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9546 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg305-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg305-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg305-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg305-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg305-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.31078 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19445 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15654 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195871 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217379 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26921 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238841 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18965 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149517 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291883 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840406 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190483 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305375 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359213 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376015 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351664 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411307 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775307 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606347 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101571 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115971 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100271 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481964 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767567 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934839 s, 11.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806207 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072013 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126725 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709737 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820074 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129332 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146779 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839741 s, 12.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8781 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9909 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (153s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 20:07:51 (1790986071) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg305-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg305-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg305-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg305-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg305-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.32288 s, 12.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150839 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131739 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728119 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173845 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172894 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234424 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236101 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25277 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22218 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197362 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222105 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193307 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399631 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412466 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418549 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.488294 s, 2.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793921 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872667 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180153 s, 58.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378272 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408894 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516368 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356803 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032183 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808169 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769178 s, 13.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362276 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107376 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443304 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102043 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797861 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613516 s, 17.1 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg305-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg305-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.091 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9729 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg305-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg305-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg305-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg305-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg305-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28959 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904277 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866843 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123799 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158455 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175136 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193121 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192757 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255068 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132711 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869227 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262147 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699583 s, 15.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197782 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356389 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292025 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333157 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635092 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815275 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815582 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074775 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665605 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654705 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083383 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082625 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921512 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824099 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789719 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105242 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100709 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851167 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842975 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103314 s, 10.1 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9565 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0168 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (151s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 20:10:22 (1790986222) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg305-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.509163 s, 33.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111937 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951926 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741627 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147323 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280257 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310328 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359422 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372195 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390743 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363952 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300338 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339716 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391921 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287974 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329193 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374059 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376429 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396987 s, 26.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789298 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536823 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555819 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985983 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598044 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415711 s, 25.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069884 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985887 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969755 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262892 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101925 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900388 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103691 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147306 s, 7.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg305-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg305-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.0177 s, 7.5 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9665 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.5@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.5@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.650483 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504294 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709239 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127472 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12996 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170365 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152643 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179349 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190632 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231101 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235573 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24395 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23532 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282627 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300321 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289237 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351989 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905522 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159931 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892419 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176227 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794318 s, 13.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788687 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12144 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159238 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917583 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196272 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202865 s, 5.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178306 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108745 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173979 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113797 s, 9.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=4 removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg305-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg305-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.27778 s, 11.3 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0206 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.5@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg305-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.525102 s, 32.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733659 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745119 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671019 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124225 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196314 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228239 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249232 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296939 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276829 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304952 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319783 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357474 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332373 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36626 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352495 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.413004 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958512 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0066116 s, 159 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101941 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106741 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104193 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619883 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021625 s, 48.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648047 s, 16.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087879 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140812 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00538571 s, 195 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025211 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022473 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247119 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117711 s, 89.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301254 s, 34.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.64119 s, 12.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.20803 s, 20.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg305-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.595451 s, 28.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602539 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841318 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736881 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202721 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202647 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1906 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237845 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191382 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148137 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185629 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186944 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17959 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212796 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270142 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239712 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246486 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02395 s, 43.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427919 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122791 s, 85.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219069 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510428 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295578 s, 35.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832081 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147177 s, 71.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499605 s, 21.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379465 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00957618 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00884616 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109925 s, 95.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0028567 s, 367 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628704 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024641 s, 42.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg305-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg305-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.2251 s, 9.3 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99854 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.8775 s, 11.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87648 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg305-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.595627 s, 28.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818568 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772545 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527366 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152688 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104329 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171475 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183999 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218877 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201894 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218847 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274937 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261841 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3008 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291455 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303865 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400537 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577423 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627728 s, 16.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657281 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989266 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634602 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638977 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712039 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484394 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086925 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882591 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128159 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940008 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103565 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937494 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1505 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129682 s, 8.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg305-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg305-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9907 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6067 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg305-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.0946 s, 15.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811704 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167945 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226498 s, 4.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14096 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127436 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110862 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162023 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26776 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204623 s, 5.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151169 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.496624 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365469 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.494323 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.559523 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.546075 s, 1.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.547117 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520495 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100928 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104693 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104565 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338555 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131219 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920114 s, 11.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102964 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932857 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828732 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772567 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106653 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10289 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984103 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931892 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109567 s, 9.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg305-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg305-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.63829 s, 10.9 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92675 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg305-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.726038 s, 23.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10357 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133408 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103253 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122508 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441653 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122406 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596941 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24459 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146528 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148814 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400641 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394797 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374031 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.427566 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328013 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.451394 s, 2.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683451 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619704 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10115 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761309 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625924 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595215 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980123 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100417 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100799 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960261 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075547 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110412 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995412 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127816 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139157 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105666 s, 9.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.2084 s, 8.6 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99546 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.2572 s, 9.3 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0083 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (442s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 20:17:44 (1790986664) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg305-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.621586 s, 27.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852216 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924269 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725449 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147556 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215579 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247958 s, 4.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295828 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219784 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293526 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302767 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362754 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335398 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330467 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37003 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302323 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809026 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591357 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342998 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664986 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546695 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686519 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665671 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870916 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722661 s, 14.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672943 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114693 s, 9.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684005 s, 15.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706848 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752852 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580191 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637115 s, 16.5 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg305-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg305-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.47028 s, 11.1 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94513 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (55s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 20:18:39 (1790986719) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.5@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.5@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.501443 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947104 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111567 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874389 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187991 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13648 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179775 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119396 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164927 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193513 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194389 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232948 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211767 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229819 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227241 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249941 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294055 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120204 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142586 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124193 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136879 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127563 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118696 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130172 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127534 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131019 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155668 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161179 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140161 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147603 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149023 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166295 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17297 s, 6.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=3 removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.1411 s, 12.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87903 s, 10.6 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (53s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 20:19:32 (1790986772) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.5@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg305-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.686553 s, 24.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794282 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130949 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918725 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168966 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160247 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221151 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187867 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653814 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195295 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285974 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340925 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196711 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306815 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365821 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315529 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221921 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665941 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734804 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751993 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771251 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859738 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126653 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737375 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925545 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108312 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101657 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105271 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114087 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117343 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116097 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110433 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137307 s, 7.6 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.92602 s, 11.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.4182 s, 19.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (49s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 20:20:21 (1790986821) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg305-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.707763 s, 23.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949062 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141202 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201036 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189352 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167979 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199979 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184596 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249359 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294065 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239296 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282129 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212282 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293262 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250547 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375973 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322352 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330615 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348233 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503075 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566598 s, 18.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627445 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762673 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597667 s, 17.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056815 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638375 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712512 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768058 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758917 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113986 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127128 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893308 s, 11.7 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.3903 s, 9.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91507 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.10513 s, 11.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98516 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (102s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 20:22:03 (1790986923) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg305-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.11406 s, 15.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698432 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133305 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14437 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118426 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111791 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148584 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500997 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225107 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224155 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272447 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223874 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124408 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318393 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21095 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296174 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.43855 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105571 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113682 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887837 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130196 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865383 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145317 s, 7.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14323 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932071 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116459 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886658 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116746 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847679 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812743 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962294 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133548 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100306 s, 10.5 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9587 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0175 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (79s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 20:23:22 (1790987002) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg305-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.7053 s, 23.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476691 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916662 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100842 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09321 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121476 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10734 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610444 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445979 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068064 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110931 s, 9.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8586d800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248677 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222892 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228767 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267453 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287136 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333412 s, 3.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8586d800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171052 s, 61.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943891 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694721 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419588 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842633 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00547 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423895 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906009 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480128 s, 21.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353488 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134017 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083705 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0043671 s, 240 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00233395 s, 449 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893479 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339568 s, 30.9 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.89615 s, 11.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92573 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg305-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.633971 s, 26.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125743 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123579 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324385 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249311 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320252 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290498 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31525 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35637 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350598 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32218 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359323 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335075 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400766 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317139 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396755 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420151 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650661 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395461 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102348 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425788 s, 24.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528646 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824925 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066093 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112134 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903648 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716628 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103213 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783433 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799554 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882527 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952171 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10031 s, 10.5 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.2585 s, 7.9 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92006 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.3251 s, 9.3 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0417 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (147s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 20:25:49 (1790987149) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg305-client.virtnet /mnt/lustre (opts:) Stopping client oleg305-client.virtnet /mnt/lustre opts: Stopping clients: oleg305-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg305-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg305-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg305-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg305-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg305-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.5,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg305-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.5,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg305-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.5,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/71298/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.5,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg305-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing load_modules_local oleg305-server: kmemleak disabled oleg305-server: Loading modules from /home/green/git/lustre-release/lustre oleg305-server: detected 4 online CPUs by sysfs oleg305-server: MODOPTS_LIBCFS= oleg305-server: Force libcfs to create 2 CPU partitions oleg305-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre Starting client oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre Started clients oleg305-client.virtnet: 192.168.203.105@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 Starting client oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 Started clients oleg305-client.virtnet: 192.168.203.105@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9e3d82c4b800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9e3d8828d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e3d82c4b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e3d8828d800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.598566 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893976 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109543 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722874 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100702 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847801 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259018 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293087 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346397 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355984 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354674 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344642 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270504 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260443 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313236 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.453213 s, 2.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793656 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494223 s, 21.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658176 s, 15.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032378 s, 32.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981121 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943335 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10789 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775687 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065287 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791017 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619867 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918908 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776267 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120387 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757756 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880068 s, 11.9 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4988 s, 10.0 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.41073 s, 19.4 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.815757 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456398 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761968 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934453 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103913 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203588 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213304 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261489 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325424 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353249 s, 3.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306582 s, 3.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299054 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307407 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325601 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365833 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319058 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.440782 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169228 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249511 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158142 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158805 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177765 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191063 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181989 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191762 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190957 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196659 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18132 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202071 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207462 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193361 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198867 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219174 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8828d800.lock_unused_count=1 removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.78145 s, 11.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97539 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (203s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 20:29:12 (1790987352) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg305-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (17s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 20:29:30 (1790987370) On MGS 192.168.203.105, active = nodemap.active=1 On MGS 192.168.203.105, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.635484 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641258 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824726 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121774 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100008 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1437 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120943 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1938 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394546 s, 26.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198437 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167456 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183378 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222563 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142437 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18606 s, 5.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172354 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605894 s, 17.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8828d800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8828d800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144279 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131584 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147011 s, 7.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21815 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178217 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175453 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245324 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239594 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262081 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217817 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264467 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184725 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264992 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261858 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266976 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214343 s, 4.9 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg305-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg305-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.76708 s, 12.0 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.34521 s, 19.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.203.105, TBF.id = On MGS 192.168.203.105, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.735942 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833453 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616402 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125031 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666998 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134565 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115237 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113645 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239642 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142911 s, 7.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8828d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239873 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.395039 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39544 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410494 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425872 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429927 s, 2.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8828d800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41116 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728671 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408785 s, 25.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684896 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809287 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952109 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514546 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046319 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644798 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618878 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735275 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602229 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715541 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830966 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871454 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688587 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894286 s, 11.7 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg305-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg305-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.29896 s, 12.6 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.2656 s, 32.1 MB/s Read runtime is 5 s, speed is 20.000000 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg305-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg305-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 On MGS 192.168.203.105, active = nodemap.active=1 On MGS 192.168.203.105, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.457799 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657461 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928988 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186715 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123419 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171076 s, 6.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1476 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170012 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191912 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131034 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185066 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199575 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237092 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23842 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226109 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190155 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255282 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472652 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425209 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565572 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615503 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705058 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403412 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045559 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573092 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439201 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638144 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640104 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903787 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071375 s, 14.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998509 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678683 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786648 s, 13.3 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg305-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg305-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.99931 s, 13.1 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92779 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.203.105, TBF.id = On MGS 192.168.203.105, active = nodemap.active=0 PASS 77kj (172s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 20:32:22 (1790987542) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (10s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 20:32:32 (1790987552) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg305-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.83638 s, 273 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg305-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0511 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (56s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 20:33:29 (1790987609) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.425273 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077411 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10505 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116934 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087241 s, 12.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8828d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181679 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184513 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136209 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256692 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233234 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172854 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247682 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290568 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239704 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222924 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279905 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284317 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8828d800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8828d800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530856 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959017 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647498 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104523 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676405 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518677 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768788 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937875 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087401 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818772 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921511 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682896 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977711 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992111 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960076 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154372 s, 6.8 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg305-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg305-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.19763 s, 14.6 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.11214 s, 20.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.516449 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345354 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993733 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100298 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092729 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104595 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188148 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24526 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260925 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304085 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310785 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320605 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278732 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279325 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321269 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306573 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280958 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e3d8828d800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9e3d8828d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762853 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817652 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118136 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136869 s, 7.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954952 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120078 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953965 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967438 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136014 s, 7.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101083 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12423 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105078 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116767 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115236 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114429 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126814 s, 8.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg305-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg305-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.01999 s, 14.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.12218 s, 20.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (85s) == sanityn test 77o: Changing rank should not panic ====== 20:34:53 (1790987693) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (12s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 20:35:06 (1790987706) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (104s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 20:36:50 (1790987810) mds.MDS.mdt.nrs_policies=tbf oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg305-client: oleg305-server: ssh exited with exit code 36 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg305-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (32s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 20:37:22 (1790987842) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273196 kill md_thread xargs PASS 77r (50s) == sanityn test 77s: Check TBF LRU shrinker ============== 20:38:12 (1790987892) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (48s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 20:39:00 (1790987940) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (7s) == sanityn test 79: xattr: intent error ================== 20:39:07 (1790987947) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 2 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935727 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 20:39:09.000000000 -0400 Modify: 2026-10-02 20:39:09.000000000 -0400 Change: 2026-10-02 20:39:09.000000000 -0400 Birth: 2026-10-02 20:39:09.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (16s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 20:39:23 (1790987963) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 20:39:26 (1790987966) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 20:39:29 (1790987969) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (2s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 20:39:31 (1790987971) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 20:39:34 (1790987974) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 20:39:37 (1790987977) - open/close 737 (time 1790987991.56 total 10.01 last 73.66) - open/close 1547 (time 1790988001.57 total 20.02 last 80.92) - open/close 2322 (time 1790988011.57 total 30.02 last 77.45) total: 2500 open/close in 32.42 seconds: 77.11 ops/second snapshot_time 1790988083.863027854 secs.nsecs start_time 1790987979.863257362 secs.nsecs elapsed_time 103.999770492 secs.nsecs open 2505 samples [usecs] 134 26734 4279370 10791100702 close 2504 samples [usecs] 172 7733 1157688 982095312 mknod 2501 samples [usecs] 689 26616 3792273 8708488105 unlink 2500 samples [usecs] 814 64557 5908940 54576952602 mkdir 2 samples [usecs] 2321 6623 8944 49251170 rmdir 2 samples [usecs] 16033 16244 32277 520924625 rename 2500 samples [usecs] 524 38631 13626029 130052828689 getattr 3762 samples [usecs] 7 5832 191292 137098162 parallel_rename_file 2500 samples [usecs] 524 38631 13626029 130052828689 crossdir_rename 2500 samples [usecs] 524 38631 13626029 130052828689 readpage 32 samples [usecs] 4874 38090 604273 14048083329 PASS 81d (110s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 20:41:27 (1790988087) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7617 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7617 PASS 82 (6s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 20:41:33 (1790988093) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (2s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 20:41:35 (1790988095) fail_loc=0x8000060b PASS 84 (10s) == sanityn test 85: Lustre API root cache race =========== 20:41:45 (1790988105) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1790988106 Finishing test test1 at 1790988109 PASS 85 (8s) == sanityn test 90: open/create and unlink striped directory ========================================================== 20:41:53 (1790988113) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 20:41:56 (1790988116) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (2s) == sanityn test 92: create remote directory under orphan directory ========================================================== 20:41:59 (1790988119) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 20:42:01 (1790988121) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4855 0x12f7 0x240000400 1 4814 0x12ce 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 4815 0x12cf 0x280000400 0 4856 0x12f8 0x240000400 rc1=0 and rc2=0 PASS 93 (13s) == sanityn test 94: signal vs CP callback race =========== 20:42:14 (1790988134) osc.lustre-OST0000-osc-ffff9e3d82c4b800.idle_timeout=0 osc.lustre-OST0000-osc-ffff9e3d8828d800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9e3d82c4b800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9e3d8828d800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.166561 s, 49.2 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.00124 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6395: 290406 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0408154 s, 100 kB/s osc.lustre-OST0000-osc-ffff9e3d82c4b800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9e3d8828d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e3d82c4b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e3d8828d800.idle_timeout=debug PASS 94 (13s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 20:42:27 (1790988147) llite.lustre-ffff9e3d82c4b800.fast_read=0 llite.lustre-ffff9e3d8828d800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0422754 s, 49.6 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9e3d82c4b800.fast_read=1 llite.lustre-ffff9e3d8828d800.fast_read=1 PASS 95a (18s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 20:42:45 (1790988165) llite.lustre-ffff9e3d82c4b800.fast_read=0 llite.lustre-ffff9e3d8828d800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00548096 s, 2.2 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9e3d82c4b800.fast_read=1 llite.lustre-ffff9e3d8828d800.fast_read=1 PASS 95b (11s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 20:42:56 (1790988176) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (2s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 20:42:58 (1790988178) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00879374 s, 466 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (8s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 20:43:07 (1790988187) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.366143 s, 5.7 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (11s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 20:43:18 (1790988198) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.241706 s, 8.7 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (14s) == sanityn test 100e: DoM: read on open and file size ==== 20:43:32 (1790988212) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00886526 s, 1.4 MB/s PASS 100e (11s) == sanityn test 101a: Discard DoM data on unlink ========= 20:43:44 (1790988224) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0143454 s, 286 kB/s PASS 101a (8s) == sanityn test 101b: Discard DoM data on rename ========= 20:43:52 (1790988232) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0139764 s, 293 kB/s PASS 101b (8s) == sanityn test 101c: Discard DoM data on close-unlink === 20:44:00 (1790988240) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0254478 s, 161 kB/s PASS 101c (7s) == sanityn test 102a: Test open by handle of unlinked file ========================================================== 20:44:08 (1790988248) file: /mnt/lustre/f102a.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 90 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf90:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102a (8s) == sanityn test 102b: open by handle of a dir inside an encrypted dir ========================================================== 20:44:15 (1790988255) 192.168.203.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg305-client.virtnet /mnt/lustre2 (opts:) 192.168.203.105@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg305-client.virtnet /mnt/lustre (opts:) Starting client: oleg305-client.virtnet: -o user_xattr,flock,test_dummy_encryption 192.168.203.105@tcp:/lustre /mnt/lustre Starting client: oleg305-client.virtnet: -o user_xattr,flock,test_dummy_encryption 192.168.203.105@tcp:/lustre /mnt/lustre2 file: /mnt/lustre/d102b.sanityn/encdir/sub fh_bytes: 32 fh_type: 151 fh_data: d3 0b 00 00 02 00 00 00 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd3:0x3:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! 192.168.203.105@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,test_dummy_encryption=%s,test_dummy_encryption,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg305-client.virtnet /mnt/lustre (opts:) Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre 192.168.203.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,test_dummy_encryption=%s,test_dummy_encryption,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg305-client.virtnet /mnt/lustre2 (opts:) Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 PASS 102b (19s) == sanityn test 103: Test size correctness with lockahead ========================================================== 20:44:34 (1790988274) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1790988277 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1790988280 Finishing test test23 at 1790988281 PASS 103 (14s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 20:44:48 (1790988288) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 20:44:51 (1790988291) fail_loc=0x416 PASS 105 (18s) == sanityn test 106a: Verify the btime via statx() ======= 20:45:09 (1790988309) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 20:45:12 (1790988312) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0324005 s, 31.6 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 11 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238877265927 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 20:45:13.000000000 -0400 Modify: 2026-10-02 20:45:14.000000000 -0400 Change: 2026-10-02 20:45:14.000000000 -0400 Birth: 2026-10-02 20:45:14.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238877265927 -rw-r--r-- 1 0 0 1790988314 1790988313 1790988314 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 11 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238877265927 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 20:45:13.000000000 -0400 Modify: 2026-10-02 20:45:14.000000000 -0400 Change: 2026-10-02 20:45:14.000000000 -0400 Birth: 2026-10-02 20:45:14.000000000 -0400 1790988314 1024 11 PASS 106b (8s) == sanityn test 106c: Verify statx attributes mask ======= 20:45:20 (1790988320) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 20:45:28 (1790988328) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.290422 s, 36.1 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 4822 0x12d6 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7617 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7617 First grouplock blocks second one PASS 107a (10s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 20:45:38 (1790988338) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.323485 s, 32.4 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4865 0x1301 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7617 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (14s) == sanityn test 108a: lseek: parallel updates ============ 20:45:52 (1790988352) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00369497 s, 2.7 MB/s llite.lustre-ffff9e3d98334000.hybrid_io=0 llite.lustre-ffff9e3d98aff000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.93616 s, 1.7 MB/s PASS 108a (14s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 20:46:06 (1790988366) 192.168.203.105@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg305-client.virtnet /mnt/lustre (opts:) 192.168.203.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg305-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre 192.168.203.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.105@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg305-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg305-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing load_modules_local oleg305-server: kmemleak disabled oleg305-server: Loading modules from /home/green/git/lustre-release/lustre oleg305-server: detected 4 online CPUs by sysfs oleg305-server: MODOPTS_LIBCFS= oleg305-server: Force libcfs to create 2 CPU partitions oleg305-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre 192.168.203.105@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg305-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg305-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing load_modules_local oleg305-server: kmemleak disabled oleg305-server: Loading modules from /home/green/git/lustre-release/lustre oleg305-server: detected 4 online CPUs by sysfs oleg305-server: MODOPTS_LIBCFS= oleg305-server: Force libcfs to create 2 CPU partitions oleg305-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 PASS 109 (66s) == sanityn test 110: do not grant another lock on resend ========================================================== 20:47:12 (1790988432) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238910820355 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 20:47:13.000000000 -0400 Modify: 2026-10-02 20:47:13.000000000 -0400 Change: 2026-10-02 20:47:13.000000000 -0400 Birth: 2026-10-02 20:47:14.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg305-client.virtnet oleg305-client.virtnet PASS 110 (167s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 20:50:00 (1790988600) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (2s) == sanityn test 112: update max-inherit in default LMV === 20:50:02 (1790988602) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (3s) == sanityn test 113: check servers of specified fs ======= 20:50:05 (1790988605) lustre-OST0000-osc-ffff9e3d852f3000 active. lustre-OST0001-osc-ffff9e3d852f3000 active. lustre-OST0000-osc-ffff9e3d852f3000 active. lustre-OST0001-osc-ffff9e3d852f3000 active. MGC192.168.203.105@tcp active. PASS 113 (5s) == sanityn test 114: implicit default LMV inherit ======== 20:50:10 (1790988610) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (2s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 20:50:12 (1790988612) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 20:50:15 (1790988615) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (2s) == sanityn test 121: trunc append race =================== 20:50:17 (1790988617) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0139665 s, 1.5 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0690709 s, 297 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (7s) == sanityn test 122: directory size is consistent across mounts ========================================================== 20:50:24 (1790988624) PASS 122 (6s) == sanityn test 200: service remains healthy while able to process request ========================================================== 20:50:30 (1790988630) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.079 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0746 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1516 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1902 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1424 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1499 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1807 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1442 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.179 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1736 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1597 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1506 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.207 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1696 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2859 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3144 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2881 s, 0.1 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2724 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg305-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e3d852f3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e3d852f3000.ost_server_uuid in FULL state after 0 sec PASS 200 (75s) cleanup: ====================================================== == sanityn test complete, duration 10138 sec ============= 20:51:47 (1790988707) === sanityn: start cleanup 20:51:48 (1790988708) === Stopping clients: oleg305-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg305-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 20:55:08 (1790988908) ===