-----============= acceptance-small: sanityn ============----- Mon Sep 7 19:14:28 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg154-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 19:14:47 (1788822887) === Starting client oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre2 Mount client oleg154-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre2 Started clients oleg154-client.virtnet: 192.168.201.154@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg154-client.virtnet: executing check_config_client /mnt/lustre oleg154-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg154-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88d8d124d000.idle_timeout=debug osc.lustre-OST0000-osc-ffff88d8e0eea800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88d8d124d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88d8e0eea800.idle_timeout=debug disable quota as required oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 19:15:18 (1788822918) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255396 s, 41.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7614] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 19:15:20 (1788822920) PASS 0 (7s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 19:15:27 (1788822927) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (5s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 19:15:33 (1788822933) -rw-r--r-- 1 root root 0 Sep 7 19:15 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (6s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 19:15:39 (1788822939) -rw-r--r-- 1 root root 0 Sep 7 19:15 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (7s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 19:15:45 (1788822945) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 19:15:51 (1788822951) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (6s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 19:15:57 (1788822957) total 1 -rwxrwxrwx 1 root root 0 Sep 7 19:15 f2a -rwxrwxrwx 1 root root 0 Sep 7 19:15 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 7 19:15 f2a -rwxrwxrwx 1 root root 0 Sep 7 19:15 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0328002 s, 15.6 kB/s PASS 2e (7s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 19:16:05 (1788822965) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 19:16:08 (1788822968) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.422285 s, 5.0 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (9s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 19:16:17 (1788822977) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 19:16:24 (1788822984) PASS 4 (7s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 19:16:31 (1788822991) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (8s) == sanityn test 6: remove of open file on other node ============================================================================ 19:16:39 (1788822999) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 19:16:46 (1788823006) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (8s) == sanityn test 8: remove of open special file on other node ==================================================================== 19:16:54 (1788823014) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (6s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 19:17:00 (1788823020) PASS 9a (9s) == sanityn test 9b: append to striped sparse file ======== 19:17:09 (1788823029) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802304 s, 13.1 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0329765 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (8s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 19:17:17 (1788823037) 1+0 records in 1+0 records out 1 byte copied, 0.0119723 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0151925 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0207817 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0081726 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0115658 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00821329 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00920482 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0103766 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0325158 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0114012 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0216468 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00964413 s, 0.1 kB/s PASS 10a (7s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 19:17:24 (1788823044) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0241423 s, 127 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0393502 s, 104 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000584054 s, 5.3 MB/s PASS 10b (8s) == sanityn test 11: execution of file opened for write should return error ============================================================== 19:17:32 (1788823052) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (8s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 19:17:40 (1788823060) warning: '-runtime' deprecated, use '-t runtime' instead - link 328 (time 1788823073.36 total 10.03 last 32.70) - link 648 (time 1788823083.37 total 20.03 last 31.99) - link 978 (time 1788823093.38 total 30.05 last 32.94) - link 1255 (time 1788823103.39 total 40.06 last 27.68) - link 1505 (time 1788823113.43 total 50.09 last 24.91) - link 1741 (time 1788823123.43 total 60.10 last 23.58) - link 1915 (time 1788823133.45 total 70.11 last 17.37) - link 2176 (time 1788823143.45 total 80.12 last 26.10) - link 2433 (time 1788823153.45 total 90.12 last 25.69) total: 2733 link in 100.01 seconds: 27.33 ops/second using seed 41339884 running for 100 seconds - stat 5658 (time 1788823075 ; total 11 ; last 11) - stat 10000 (time 1788823083 ; total 19 ; last 8) - stat 16018 (time 1788823094 ; total 30 ; last 11) - stat 20000 (time 1788823102 ; total 38 ; last 8) - stat 24140 (time 1788823113 ; total 49 ; last 11) - stat 26362 (time 1788823124 ; total 60 ; last 11) - stat 27785 (time 1788823135 ; total 71 ; last 11) - stat 30000 (time 1788823143 ; total 79 ; last 8) - stat 32218 (time 1788823154 ; total 90 ; last 11) total: 35027 stats in 100 seconds: 350.269989 stats/second - unlinked 0 (time 1788823114 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2733) error: No such file or directory total: 2732 unlinks in 86 seconds: 31.767443 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21417) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21419) - No such process PASS 12 (147s) == sanityn test 13: test directory page revocation ======= 19:20:07 (1788823207) f13 f13 PASS 13 (10s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 19:20:17 (1788823217) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22606 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 19:20:24 (1788823224) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23185 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 19:20:32 (1788823232) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23764 Terminated $DIR1/$tdir/sleep 60 PASS 14b (7s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 19:20:39 (1788823239) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24346 Terminated $DIR1/$tdir/sleep 60 PASS 14c (9s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 19:20:48 (1788823248) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24928 Terminated $DIR1/$tdir/sleep 60 PASS 14d (9s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 19:20:57 (1788823257) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7529472 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7529472KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (60s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 19:21:57 (1788823317) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.589913 s, 14.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.305956 s, 27.4 MB/s Chance of close/open is 1 in 50 Seed set to 3322 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000005[0] 1788823322.789758 mapwrite 0x7f6db0 thru 0x800e80 (0xa0d1 bytes) truncating to largest ever: 0x2b84ab 000007[1] 1788823322.927876 trunc from 0x800e81 to 0x2b84aa (0x5489d6 bytes) 000008[0] 1788823322.980215 read 0x092ee3 thru 0x0991c1 (0x62df bytes) truncating to largest ever: 0x580164 000009[1] 1788823323.006085 trunc from 0x2b84ab to 0x580163 (0x2c7cb9 bytes) 000010[1] 1788823323.056873 mapread 0x4b4b49 thru 0x4b5910 (0xdc8 bytes) truncating to largest ever: 0x716699 000014[1] 1788823323.161909 trunc from 0x580164 to 0x716698 (0x196535 bytes) 000017[0] 1788823323.181603 read 0x710ac4 thru 0x716698 (0x5bd5 bytes) 000018[1] 1788823323.213978 mapread 0x2492a2 thru 0x24cb44 (0x38a3 bytes) 000019[0] 1788823323.253548 write 0x286e79 thru 0x28bf25 (0x50ad bytes) 000021[0] 1788823323.293134 punch from 0x22c9ed to 0x233072 (0x6686 bytes) 000024[0] 1788823323.315938 punch from 0x6ef010 to 0x6fcdab (0xdd9c bytes) 000025[0] 1788823323.326631 write 0x4ed2e3 thru 0x4f0bb6 (0x38d4 bytes) 000027[1] 1788823323.352900 mapwrite 0x759521 thru 0x7666fc (0xd1dc bytes) 000030[0] 1788823323.514834 punch from 0x43dccc to 0x44c42e (0xe763 bytes) 000031[1] 1788823323.557599 punch from 0x37dd44 to 0x37e517 (0x7d4 bytes) 000034[1] 1788823323.619444 punch from 0x013629 to 0x01bb67 (0x853f bytes) 000035[1] 1788823323.630573 punch from 0x520631 to 0x520d9f (0x76f bytes) 000038[0] 1788823323.718134 mapread 0x4a74d2 thru 0x4af702 (0x8231 bytes) truncating to largest ever: 0x9fd5df 000040[0] 1788823323.794143 trunc from 0x7666fd to 0x9fd5de (0x296ee2 bytes) 000041[0] 1788823323.847711 mapwrite 0x0f0dfa thru 0x0f5fef (0x51f6 bytes) 000042[0] 1788823323.993968 mapwrite 0x4345a8 thru 0x43cd56 (0x87af bytes) 000043[0] 1788823324.063984 mapread 0x57f488 thru 0x5821f8 (0x2d71 bytes) 000044[1] 1788823324.107987 mapwrite 0x793bde thru 0x7988f6 (0x4d19 bytes) 000045[0] 1788823324.207104 trunc from 0x9fd5df to 0x10d631 (0x8effad bytes) 000047[1] 1788823324.273732 trunc from 0x10d632 to 0x76f4ea (0x661eb9 bytes) 000049[0] 1788823324.345342 read 0x1560ec thru 0x158ec9 (0x2dde bytes) 000051[1] 1788823324.365486 punch from 0x4d7081 to 0x4d9aba (0x2a3a bytes) 000054[0] 1788823324.382160 trunc from 0x76f4eb to 0x96765f (0x1f8175 bytes) 000057[0] 1788823324.496191 mapwrite 0x003412 thru 0x00458a (0x1179 bytes) 000058[0] 1788823324.578371 write 0x5548ec thru 0x55afe1 (0x66f6 bytes) 000060[1] 1788823324.603200 mapwrite 0x419e26 thru 0x41a760 (0x93b bytes) 000063[1] 1788823324.727462 trunc from 0x967660 to 0x6eb77d (0x27bee2 bytes) 000065[1] 1788823324.767016 read 0x222adb thru 0x22628d (0x37b3 bytes) 000066[1] 1788823324.773284 trunc from 0x6eb77e to 0x9ceb9e (0x2e3421 bytes) 000068[0] 1788823324.817143 mapwrite 0x9b16ec thru 0x9b67d5 (0x50ea bytes) 000069[0] 1788823324.978488 write 0x30ca73 thru 0x30e4c7 (0x1a55 bytes) 000070[0] 1788823325.002075 read 0x57c5b5 thru 0x58b711 (0xf15d bytes) 000071[0] 1788823325.019846 read 0x227cad thru 0x22ee94 (0x71e8 bytes) 000072[0] 1788823325.026235 punch from 0x37277a to 0x37d4ea (0xad71 bytes) 000073[0] 1788823325.033147 write 0x35da85 thru 0x368490 (0xaa0c bytes) 000074[1] 1788823325.057542 punch from 0x06f2bc to 0x079f0f (0xac54 bytes) 000075[0] 1788823325.078634 mapread 0x78efc3 thru 0x79393a (0x4978 bytes) 000076[0] 1788823325.108846 read 0x66c0aa thru 0x66ed89 (0x2ce0 bytes) 000077[0] 1788823325.114238 mapread 0x562a72 thru 0x57070c (0xdc9b bytes) 000079[1] 1788823325.164514 punch from 0x4747c4 to 0x480a28 (0xc265 bytes) 000080[1] 1788823325.178578 read 0x28290a thru 0x2833ce (0xac5 bytes) 000081[1] 1788823325.182981 mapwrite 0x75f768 thru 0x76bdb6 (0xc64f bytes) 000085[0] 1788823325.307046 mapread 0x990717 thru 0x99206b (0x1955 bytes) 000086[0] 1788823325.346798 mapread 0x4a720c thru 0x4ada7b (0x6870 bytes) 000087[1] 1788823325.396628 punch from 0x26b749 to 0x26bc4b (0x503 bytes) 000090[0] 1788823325.415817 mapwrite 0x18a7ff thru 0x195a0a (0xb20c bytes) 000091[1] 1788823325.669015 mapread 0x0f0069 thru 0x0f8b2a (0x8ac2 bytes) 000092[1] 1788823325.699740 write 0x664819 thru 0x671648 (0xce30 bytes) 000093[0] 1788823325.729558 mapread 0x067fc1 thru 0x06abc4 (0x2c04 bytes) 000094[0] 1788823325.766756 read 0x861bee thru 0x867f7a (0x638d bytes) 000096[0] 1788823325.810878 mapread 0x08a1da thru 0x095668 (0xb48f bytes) 000098[1] 1788823325.871967 mapwrite 0x33f0d4 thru 0x341be1 (0x2b0e bytes) 000100[0] 1788823325.947077 read 0x02f3d6 thru 0x03015b (0xd86 bytes) 000103[1] 1788823325.988773 read 0x9c45f3 thru 0x9c8ca3 (0x46b1 bytes) 000106[1] 1788823326.033169 write 0x9b9416 thru 0x9bd67f (0x426a bytes) 000107[0] 1788823326.065751 punch from 0x58dbc5 to 0x593005 (0x5441 bytes) 000108[0] 1788823326.108697 mapwrite 0x47bae1 thru 0x47f6e6 (0x3c06 bytes) 000110[0] 1788823326.229867 punch from 0x8164a6 to 0x818695 (0x21f0 bytes) 000111[0] 1788823326.313747 mapwrite 0x49b88b thru 0x4a9d7b (0xe4f1 bytes) 000112[0] 1788823326.440895 mapread 0x629025 thru 0x63873a (0xf716 bytes) 000113[1] 1788823326.538776 trunc from 0x9ceb9f to 0x98c50f (0x4268f bytes) 000114[0] 1788823326.619402 mapwrite 0x561cc0 thru 0x563132 (0x1473 bytes) 000115[0] 1788823326.762209 mapwrite 0x96a016 thru 0x96d4d6 (0x34c1 bytes) 000119[0] 1788823326.968847 mapwrite 0x88c5a6 thru 0x88e0fd (0x1b58 bytes) 000120[0] 1788823327.084483 mapread 0x657f3e thru 0x6658fc (0xd9bf bytes) 000121[1] 1788823327.119134 write 0x70936e thru 0x70d6e4 (0x4377 bytes) 000124[1] 1788823327.179140 mapwrite 0x6bdeeb thru 0x6c9bab (0xbcc1 bytes) 000125[0] 1788823327.289138 write 0x2bbe59 thru 0x2c2b6c (0x6d14 bytes) 000126[0] 1788823327.318523 read 0x404d92 thru 0x409252 (0x44c1 bytes) 000127[0] 1788823327.337699 write 0x106b26 thru 0x107c14 (0x10ef bytes) 000128[1] 1788823327.346601 trunc from 0x98c510 to 0x55907b (0x433494 bytes) 000130[0] 1788823327.395656 punch from 0x443401 to 0x446882 (0x3482 bytes) 000132[0] 1788823327.442054 punch from 0x1057ed to 0x10b165 (0x5979 bytes) 000134[0] 1788823327.458999 mapread 0x23b33d thru 0x23d489 (0x214d bytes) 000135[0] 1788823327.497045 trunc from 0x55907c to 0x1705f4 (0x3e8a87 bytes) 000137[1] 1788823327.546630 mapwrite 0x6a1458 thru 0x6aa5e6 (0x918f bytes) 000141[0] 1788823327.697874 punch from 0x24b706 to 0x25a385 (0xec80 bytes) 000142[0] 1788823327.725989 trunc from 0x6aa5e7 to 0x1c4749 (0x4e5e9d bytes) 000143[0] 1788823327.766582 trunc from 0x1c474a to 0x2221b8 (0x5da6f bytes) 000144[1] 1788823327.780157 mapwrite 0x2a02b4 thru 0x2a4198 (0x3ee5 bytes) 000145[0] 1788823327.924617 trunc from 0x2a4199 to 0x9b5119 (0x710f81 bytes) 000146[1] 1788823327.988067 mapread 0x4ce28e thru 0x4de044 (0xfdb7 bytes) 000148[0] 1788823328.068723 read 0x55efab thru 0x56188c (0x28e2 bytes) 000149[0] 1788823328.091001 trunc from 0x9b511a to 0x049ffa (0x96b11f bytes) 000152[0] 1788823328.217367 mapread 0x02cb50 thru 0x02e000 (0x14b1 bytes) 000153[0] 1788823328.239872 mapwrite 0x53bbfc thru 0x53c6b4 (0xab9 bytes) 000154[0] 1788823328.332774 write 0x7cf0b2 thru 0x7d8e40 (0x9d8f bytes) 000155[1] 1788823328.335043 mapread 0x4d06c6 thru 0x4d0b94 (0x4cf bytes) 000157[0] 1788823328.427673 mapwrite 0x54bf75 thru 0x54d659 (0x16e5 bytes) 000158[1] 1788823328.504974 trunc from 0x7d8e41 to 0x559d50 (0x27f0f0 bytes) 000160[1] 1788823328.591512 mapwrite 0x9c96bb thru 0x9cc25d (0x2ba3 bytes) 000161[1] 1788823328.683593 write 0x633e66 thru 0x63ae8b (0x7026 bytes) 000162[1] 1788823328.730108 mapread 0x859f60 thru 0x863035 (0x90d6 bytes) 000163[0] 1788823328.820168 read 0x72b8c3 thru 0x7393fb (0xdb39 bytes) 000165[1] 1788823328.879688 read 0x798e5e thru 0x7a4020 (0xb1c3 bytes) 000168[0] 1788823328.923384 mapwrite 0x269543 thru 0x278bc3 (0xf681 bytes) 000171[1] 1788823329.082766 write 0x4db16f thru 0x4de5ec (0x347e bytes) 000172[0] 1788823329.116926 read 0x45b42f thru 0x45e857 (0x3429 bytes) 000174[0] 1788823329.157857 mapwrite 0x70244d thru 0x708cd1 (0x6885 bytes) 000175[1] 1788823329.273717 trunc from 0x9cc25e to 0x5dbdde (0x3f047f bytes) 000177[0] 1788823329.327880 read 0x1b93fb thru 0x1c0f56 (0x7b5c bytes) 000178[1] 1788823329.351872 read 0x18d45b thru 0x193f4b (0x6af1 bytes) 000179[1] 1788823329.359975 mapread 0x316104 thru 0x31e818 (0x8715 bytes) 000180[1] 1788823329.377841 read 0x5b84a6 thru 0x5c6c9d (0xe7f8 bytes) 000181[1] 1788823329.385977 punch from 0x48df8d to 0x4925ff (0x4673 bytes) 000182[1] 1788823329.390928 read 0x48a09f thru 0x49909d (0xefff bytes) 000183[1] 1788823329.403383 trunc from 0x5dbddf to 0x835fc6 (0x25a1e8 bytes) 000184[0] 1788823329.460753 read 0x79884d thru 0x79f528 (0x6cdc bytes) 000185[1] 1788823329.491913 trunc from 0x835fc7 to 0x207d2c (0x62e29a bytes) 000186[0] 1788823329.551712 mapread 0x045ee5 thru 0x0528f9 (0xca15 bytes) 000187[1] 1788823329.622729 mapread 0x070366 thru 0x07fe80 (0xfb1b bytes) 000192[1] 1788823329.721250 punch from 0x0d29b0 to 0x0d57c3 (0x2e14 bytes) 000195[0] 1788823329.766971 mapwrite 0x850d41 thru 0x85787d (0x6b3d bytes) 000198[1] 1788823329.940691 mapread 0x67be72 thru 0x681f36 (0x60c5 bytes) 000200[1] 1788823330.010827 punch from 0x2bae9d to 0x2c2139 (0x729d bytes) 000201[0] 1788823330.044030 write 0x6e6676 thru 0x6f1951 (0xb2dc bytes) 000202[1] 1788823330.088518 trunc from 0x85787e to 0x659f89 (0x1fd8f4 bytes) 000203[0] 1788823330.145949 read 0x33448d thru 0x33b003 (0x6b77 bytes) 000204[0] 1788823330.171360 mapread 0x60609e thru 0x610b71 (0xaad4 bytes) 000205[0] 1788823330.225880 punch from 0x2167a0 to 0x22280a (0xc06b bytes) 000206[1] 1788823330.241913 write 0x710d88 thru 0x7141d8 (0x3451 bytes) 000207[1] 1788823330.274527 mapwrite 0x3373ee thru 0x33ac77 (0x388a bytes) 000209[0] 1788823330.387759 trunc from 0x7141d9 to 0x3705fa (0x3a3bde bytes) 000211[0] 1788823330.458505 trunc from 0x3705fb to 0x35c927 (0x13cd3 bytes) 000214[0] 1788823330.559675 write 0x4d6f57 thru 0x4dd3df (0x6489 bytes) 000215[0] 1788823330.567006 mapwrite 0x01e27e thru 0x02509f (0x6e22 bytes) 000216[1] 1788823330.692836 punch from 0x46b1fd to 0x46ce51 (0x1c55 bytes) 000219[1] 1788823330.811861 trunc from 0x4dd3e0 to 0x357d9d (0x185642 bytes) 000221[1] 1788823330.889031 trunc from 0x357d9e to 0x95b4c9 (0x60372c bytes) 000223[0] 1788823330.942212 trunc from 0x95b4ca to 0x52f5f3 (0x42bed6 bytes) 000224[0] 1788823330.978489 read 0x2d2cb4 thru 0x2dafce (0x831b bytes) 000226[1] 1788823331.022545 read 0x2aaf81 thru 0x2af2ca (0x434a bytes) 000228[1] 1788823331.069197 read 0x4584ab thru 0x4604c1 (0x8017 bytes) 000230[0] 1788823331.108493 read 0x1b8a1d thru 0x1c185f (0x8e43 bytes) 000231[1] 1788823331.116666 punch from 0x233987 to 0x2397c9 (0x5e43 bytes) 000235[1] 1788823331.154892 mapread 0x083dfb thru 0x087e2d (0x4033 bytes) 000239[0] 1788823331.224910 mapwrite 0x7e04b0 thru 0x7eb0aa (0xabfb bytes) 000240[1] 1788823331.404907 write 0x8929a4 thru 0x89b40a (0x8a67 bytes) 000243[0] 1788823331.537014 mapwrite 0x40c02c thru 0x40ddbf (0x1d94 bytes) 000244[1] 1788823331.675606 trunc from 0x89b40b to 0x054dbc (0x84664e bytes) 000245[1] 1788823331.748206 punch from 0x02da96 to 0x0368c2 (0x8e2d bytes) 000246[0] 1788823331.777588 trunc from 0x054dbd to 0x47ef2a (0x42a16e bytes) 000247[1] 1788823331.848927 punch from 0x2e9811 to 0x2f9176 (0xf966 bytes) 000248[0] 1788823331.868014 punch from 0x17fcff to 0x1831c3 (0x34c5 bytes) 000250[1] 1788823331.957741 trunc from 0x47ef2b to 0x951e89 (0x4d2f5f bytes) 000252[0] 1788823332.050292 read 0x090b21 thru 0x09d1e8 (0xc6c8 bytes) 000255[1] 1788823332.108310 mapwrite 0x2c09cd thru 0x2c3984 (0x2fb8 bytes) 000260[0] 1788823332.306361 trunc from 0x951e8a to 0x9d7e27 (0x85f9e bytes) 000261[1] 1788823332.348799 mapwrite 0x438c2e thru 0x44492d (0xbd00 bytes) 000265[1] 1788823332.502765 punch from 0x016194 to 0x01d70f (0x757c bytes) 000266[0] 1788823332.517842 write 0x48120b thru 0x48fc7d (0xea73 bytes) 000271[0] 1788823332.611200 read 0x4d9f4d thru 0x4e2f25 (0x8fd9 bytes) 000273[1] 1788823332.632136 read 0x80a68b thru 0x81020e (0x5b84 bytes) 000274[0] 1788823332.645287 trunc from 0x9d7e28 to 0x23f06f (0x798db8 bytes) 000275[0] 1788823332.666554 mapwrite 0x3ec99b thru 0x3f45f7 (0x7c5d bytes) 000276[0] 1788823332.713199 mapread 0x09dfe5 thru 0x0a50dd (0x70f9 bytes) 000278[1] 1788823332.734885 write 0x0f5f25 thru 0x0f9e5e (0x3f3a bytes) 000279[1] 1788823332.759300 trunc from 0x3f45f8 to 0x0519f1 (0x3a2c06 bytes) 000280[1] 1788823332.790945 mapwrite 0x3e537d thru 0x3eb421 (0x60a5 bytes) 000281[1] 1788823332.832929 trunc from 0x3eb422 to 0x9ad08b (0x5c1c6a bytes) 000282[1] 1788823332.849930 mapwrite 0x45757d thru 0x45f0cc (0x7b50 bytes) 000283[0] 1788823332.884007 mapwrite 0x78b349 thru 0x79b341 (0xfff9 bytes) 000284[0] 1788823332.969648 read 0x18e5de thru 0x199751 (0xb174 bytes) 000285[0] 1788823332.986472 read 0x2c097c thru 0x2ceb5f (0xe1e4 bytes) 000286[1] 1788823333.012802 write 0x2f5a67 thru 0x2fb302 (0x589c bytes) 000287[1] 1788823333.123630 mapread 0x0799d0 thru 0x083bf5 (0xa226 bytes) 000289[0] 1788823333.215401 write 0x1ec901 thru 0x1f0c6e (0x436e bytes) 000294[1] 1788823333.396785 mapread 0x55b114 thru 0x55e472 (0x335f bytes) 000295[0] 1788823333.456593 punch from 0x602387 to 0x602ff7 (0xc71 bytes) 000296[0] 1788823333.499796 read 0x99a71d thru 0x99ab6b (0x44f bytes) 000297[0] 1788823333.549978 mapread 0x0c7e65 thru 0x0d2912 (0xaaae bytes) 000298[0] 1788823333.581822 read 0x1bf9f9 thru 0x1c7a76 (0x807e bytes) 000299[0] 1788823333.602082 punch from 0x1fd4be to 0x1fda5e (0x5a1 bytes) 000300[1] 1788823333.625331 mapwrite 0x02e004 thru 0x03442e (0x642b bytes) 000303[0] 1788823333.894451 read 0x69d6b5 thru 0x6ab66e (0xdfba bytes) 000307[0] 1788823334.013501 mapread 0x1cd5b6 thru 0x1ce425 (0xe70 bytes) 000310[1] 1788823334.094509 write 0x9fb148 thru 0x9fffff (0x4eb8 bytes) 000311[1] 1788823334.142743 trunc from 0xa00000 to 0x64b844 (0x3b47bb bytes) 000314[0] 1788823334.313897 mapread 0x2b1805 thru 0x2bfa7a (0xe276 bytes) 000315[1] 1788823334.430051 read 0x4201dc thru 0x426145 (0x5f6a bytes) 000316[0] 1788823334.487516 read 0x5eab68 thru 0x5f51dc (0xa675 bytes) 000317[0] 1788823334.537977 mapwrite 0x4d0a2e thru 0x4d7f95 (0x7568 bytes) 000318[0] 1788823334.756358 punch from 0x2765e1 to 0x28568b (0xf0ab bytes) 000321[0] 1788823334.955611 mapread 0x15120f thru 0x158717 (0x7509 bytes) 000323[0] 1788823335.015150 read 0x06183b thru 0x064d21 (0x34e7 bytes) 000324[1] 1788823335.046292 trunc from 0x64b845 to 0x8868df (0x23b09b bytes) 000325[0] 1788823335.116359 trunc from 0x8868e0 to 0x599c24 (0x2eccbb bytes) 000327[0] 1788823335.227294 read 0x1aaa4e thru 0x1b9906 (0xeeb9 bytes) 000328[0] 1788823335.252374 trunc from 0x599c25 to 0x92436a (0x38a746 bytes) 000329[1] 1788823335.301875 mapread 0x62dd4e thru 0x63c214 (0xe4c7 bytes) 000330[0] 1788823335.381722 write 0x35da84 thru 0x366e21 (0x939e bytes) 000332[0] 1788823335.418901 trunc from 0x92436b to 0x3331a5 (0x5f11c5 bytes) 000335[0] 1788823335.540879 read 0x193cf1 thru 0x1a21db (0xe4eb bytes) 000336[1] 1788823335.668839 write 0x3d2595 thru 0x3e0495 (0xdf01 bytes) 000337[1] 1788823335.716879 mapwrite 0x6b8a61 thru 0x6b94f0 (0xa90 bytes) 000339[0] 1788823335.870389 mapwrite 0x0cbc71 thru 0x0d7b58 (0xbee8 bytes) 000340[1] 1788823336.213762 punch from 0x020037 to 0x0263e1 (0x63ab bytes) 000341[0] 1788823336.275716 mapread 0x46b39b thru 0x479963 (0xe5c9 bytes) 000343[1] 1788823336.571857 mapread 0x393921 thru 0x39b29e (0x797e bytes) 000346[0] 1788823336.740103 read 0x3b3bd9 thru 0x3b4546 (0x96e bytes) 000348[0] 1788823336.777476 trunc from 0x6b94f1 to 0x531c8d (0x187863 bytes) 000350[1] 1788823336.930340 mapread 0x219e33 thru 0x228995 (0xeb63 bytes) 000351[0] 1788823337.072987 mapread 0x0742dc thru 0x07f89d (0xb5c2 bytes) 000353[1] 1788823337.243720 write 0x4965a3 thru 0x49e5ef (0x804d bytes) 000354[1] 1788823337.353546 punch from 0x4df6f7 to 0x4e61c3 (0x6acd bytes) 000355[0] 1788823337.401833 read 0x2e73f5 thru 0x2ebbaf (0x47bb bytes) 000358[1] 1788823337.418294 trunc from 0x531c8e to 0x37c28c (0x1b5a01 bytes) 000359[0] 1788823337.492857 read 0x1a9a22 thru 0x1b949e (0xfa7d bytes) 000361[1] 1788823337.658336 punch from 0x03e525 to 0x048d55 (0xa831 bytes) 000363[1] 1788823337.675753 mapread 0x207172 thru 0x20b8d5 (0x4764 bytes) 000364[0] 1788823337.691536 mapwrite 0x51e192 thru 0x52bddf (0xdc4e bytes) 000365[0] 1788823337.915402 mapwrite 0x63fd08 thru 0x642a49 (0x2d42 bytes) 000366[0] 1788823338.075533 read 0x230121 thru 0x2328bc (0x279c bytes) 000368[1] 1788823338.159080 read 0x471a27 thru 0x472918 (0xef2 bytes) 000369[0] 1788823338.225262 mapwrite 0x8a87ee thru 0x8b5871 (0xd084 bytes) 000371[1] 1788823338.533087 mapread 0x267e1d thru 0x26de8e (0x6072 bytes) 000372[0] 1788823338.663543 mapwrite 0x0e35cd thru 0x0f0669 (0xd09d bytes) 000373[1] 1788823338.925263 mapwrite 0x636d0e thru 0x642423 (0xb716 bytes) 000379[0] 1788823339.163478 mapwrite 0x371568 thru 0x37d527 (0xbfc0 bytes) 000381[0] 1788823339.475541 trunc from 0x8b5872 to 0x24ea8f (0x666de2 bytes) 000383[1] 1788823339.571267 punch from 0x0dc1e9 to 0x0de548 (0x2360 bytes) 000384[1] 1788823339.620304 mapread 0x003ba1 thru 0x00f8b7 (0xbd17 bytes) 000385[1] 1788823339.680438 punch from 0x13236a to 0x13b392 (0x9029 bytes) 000387[0] 1788823339.747391 punch from 0x1f1e32 to 0x201335 (0xf504 bytes) 000390[1] 1788823339.908832 trunc from 0x24ea90 to 0x593191 (0x344702 bytes) 000391[1] 1788823340.047852 trunc from 0x593192 to 0x0a83fe (0x4ead93 bytes) 000392[0] 1788823340.089530 write 0x5c211f thru 0x5c4d20 (0x2c02 bytes) 000393[0] 1788823340.151909 read 0x30e0cd thru 0x30ef14 (0xe48 bytes) 000395[1] 1788823340.220211 trunc from 0x5c4d21 to 0x9c2d73 (0x3fe053 bytes) 000398[0] 1788823340.420952 trunc from 0x9c2d74 to 0x644f94 (0x37dddf bytes) 000403[1] 1788823340.665837 punch from 0x1a5723 to 0x1af980 (0xa25e bytes) 000404[1] 1788823340.701305 mapwrite 0x325765 thru 0x32c0a1 (0x693d bytes) 000405[1] 1788823340.850820 write 0x5bd21f thru 0x5c2d87 (0x5b69 bytes) 000407[0] 1788823340.900913 mapread 0x258e2a thru 0x25fc34 (0x6e0b bytes) 000409[0] 1788823340.944540 read 0x01a43c thru 0x021a4c (0x7611 bytes) 000410[0] 1788823341.109892 write 0x516541 thru 0x522aa4 (0xc564 bytes) 000411[0] 1788823341.214055 punch from 0x545fe4 to 0x54926c (0x3289 bytes) 000412[1] 1788823341.273239 mapread 0x046681 thru 0x047bec (0x156c bytes) 000416[0] 1788823341.335409 mapwrite 0x3b6744 thru 0x3b6934 (0x1f1 bytes) 000417[1] 1788823341.572244 trunc from 0x644f95 to 0x966a19 (0x321a85 bytes) 000418[0] 1788823341.637717 read 0x19cb38 thru 0x1a804c (0xb515 bytes) 000419[1] 1788823341.689543 trunc from 0x966a1a to 0x2c8b9e (0x69de7b bytes) 000422[1] 1788823341.781752 mapread 0x151b9d thru 0x156d50 (0x51b4 bytes) 000425[0] 1788823341.825352 punch from 0x01e59d to 0x029978 (0xb3dc bytes) 000426[1] 1788823341.850366 read 0x090868 thru 0x09885d (0x7ff6 bytes) 000428[0] 1788823341.907508 mapread 0x0a936f thru 0x0b229d (0x8f2f bytes) 000431[0] 1788823341.949732 write 0x7138bd thru 0x721c60 (0xe3a4 bytes) 000432[1] 1788823341.965297 read 0x1df903 thru 0x1e052b (0xc29 bytes) 000433[1] 1788823341.981799 punch from 0x1765e3 to 0x17f7cc (0x91ea bytes) 000434[1] 1788823342.032287 read 0x720c21 thru 0x7210c5 (0x4a5 bytes) 000435[0] 1788823342.072901 punch from 0x5e5cd2 to 0x5e715a (0x1489 bytes) 000436[0] 1788823342.110069 mapwrite 0x2fd4e6 thru 0x304401 (0x6f1c bytes) 000437[1] 1788823342.214709 write 0x9455ea thru 0x94e637 (0x904e bytes) 000438[1] 1788823342.242840 write 0x1a81aa thru 0x1b3ffe (0xbe55 bytes) 000439[0] 1788823342.267029 read 0x3ce89e thru 0x3d5dd9 (0x753c bytes) 000440[0] 1788823342.314903 punch from 0x81ed39 to 0x82589a (0x6b62 bytes) 000441[1] 1788823342.333268 read 0x76893d thru 0x7690eb (0x7af bytes) 000444[1] 1788823342.407482 mapread 0x81be86 thru 0x82b905 (0xfa80 bytes) 000446[0] 1788823342.453326 write 0x8710ad thru 0x88102a (0xff7e bytes) 000447[0] 1788823342.488358 mapread 0x7c8ffa thru 0x7cc504 (0x350b bytes) 000448[1] 1788823342.526551 mapwrite 0x5ddc70 thru 0x5ddc82 (0x13 bytes) 000452[1] 1788823342.715682 mapread 0x88ed5b thru 0x891324 (0x25ca bytes) 000453[0] 1788823342.759501 write 0x23378c thru 0x23e4a5 (0xad1a bytes) 000454[0] 1788823342.802744 mapwrite 0x5eb1c6 thru 0x5f9e5a (0xec95 bytes) 000456[1] 1788823342.988750 trunc from 0x94e638 to 0x8476ed (0x106f4a bytes) 000457[0] 1788823343.084362 punch from 0x5d2609 to 0x5dab1f (0x8517 bytes) 000458[1] 1788823343.128923 trunc from 0x8476ee to 0x54e093 (0x2f965a bytes) 000462[0] 1788823343.236180 write 0x547408 thru 0x54e2d9 (0x6ed2 bytes) 000466[1] 1788823343.309242 trunc from 0x54e2da to 0x44b20e (0x1030cb bytes) 000467[0] 1788823343.387520 punch from 0x212075 to 0x219b68 (0x7af4 bytes) 000468[1] 1788823343.417148 write 0x8172b6 thru 0x824c3f (0xd98a bytes) 000469[0] 1788823343.451701 write 0x2cc85d thru 0x2d2083 (0x5827 bytes) 000470[1] 1788823343.496237 mapwrite 0x688617 thru 0x6910e5 (0x8acf bytes) 000471[1] 1788823343.792729 write 0x6171f0 thru 0x61e3e4 (0x71f5 bytes) 000476[0] 1788823343.928228 mapread 0x59d347 thru 0x5abe74 (0xeb2e bytes) 000477[0] 1788823343.978895 write 0x5f6795 thru 0x603def (0xd65b bytes) 000479[1] 1788823344.058790 punch from 0x756527 to 0x76182c (0xb306 bytes) 000482[0] 1788823344.125486 read 0x05b196 thru 0x06accd (0xfb38 bytes) 000485[1] 1788823344.183535 mapwrite 0x10d71e thru 0x114029 (0x690c bytes) 000486[1] 1788823344.279510 read 0x779049 thru 0x77990c (0x8c4 bytes) 000487[0] 1788823344.312472 write 0x60cb20 thru 0x611dab (0x528c bytes) 000488[0] 1788823344.362965 trunc from 0x824c40 to 0x676a5a (0x1ae1e5 bytes) 000489[0] 1788823344.437888 trunc from 0x676a5b to 0x4f31c0 (0x18389a bytes) 000490[1] 1788823344.493238 mapwrite 0x82efc0 thru 0x83ca73 (0xdab4 bytes) 000491[1] 1788823344.627606 read 0x6af939 thru 0x6b7ffc (0x86c4 bytes) 000492[1] 1788823344.669530 trunc from 0x83ca74 to 0x988d0d (0x14c29a bytes) 000493[1] 1788823344.719092 mapwrite 0x9718cb thru 0x98182d (0xff63 bytes) 000495[0] 1788823344.858585 mapread 0x198394 thru 0x1a2550 (0xa1bd bytes) 000496[0] 1788823344.935862 mapread 0x532846 thru 0x535c77 (0x3432 bytes) 000497[1] 1788823345.008141 write 0x99aa43 thru 0x99b619 (0xbd7 bytes) 000498[1] 1788823345.046261 punch from 0x187ef0 to 0x191de1 (0x9ef2 bytes) 000499[1] 1788823345.068509 write 0x929b42 thru 0x939791 (0xfc50 bytes) 000500[0] 1788823345.102100 trunc from 0x99b61a to 0x15f9ea (0x83bc2f bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 3345 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003[1] 1788823346.302961 write 0x0dc000 thru 0x0dc86d (0x86e bytes) truncating to largest ever: 0x28b648 000004[0] 1788823346.349043 trunc from 0x0dc86e to 0x28b647 (0x1aedda bytes) 000006[0] 1788823346.415088 write 0x385000 thru 0x38914b (0x414c bytes) 000007[1] 1788823346.448033 mapread 0x116000 thru 0x11a9a0 (0x49a1 bytes) 000008[0] 1788823346.524126 write 0x200000 thru 0x20aceb (0xacec bytes) 000009[1] 1788823346.567459 mapread 0x151000 thru 0x15ee04 (0xde05 bytes) 000010[0] 1788823346.630965 read 0x038000 thru 0x045b35 (0xdb36 bytes) 000016[0] 1788823346.650672 punch from 0x281d90 to 0x287310 (0x5581 bytes) 000017[0] 1788823346.675961 mapread 0x0a7000 thru 0x0ac543 (0x5544 bytes) 000019[1] 1788823346.725166 punch from 0x350c6f to 0x35e2bf (0xd651 bytes) 000020[0] 1788823346.772414 punch from 0x10fa15 to 0x11d2b5 (0xd8a1 bytes) 000021[1] 1788823346.790647 mapread 0x383000 thru 0x387ece (0x4ecf bytes) truncating to largest ever: 0x29be62 000023[1] 1788823346.846166 trunc from 0x38914c to 0x29be61 (0xed2ea bytes) 000024[0] 1788823346.972073 punch from 0x21cb23 to 0x22c471 (0xf94f bytes) 000028[1] 1788823347.045826 read 0x20e000 thru 0x20f910 (0x1911 bytes) 000029[0] 1788823347.065592 mapwrite 0x94f000 thru 0x9504d4 (0x14d5 bytes) 000030[1] 1788823347.235239 punch from 0x68ed8b to 0x691288 (0x24fe bytes) 000031[0] 1788823347.259275 mapwrite 0x4aa000 thru 0x4b34d3 (0x94d4 bytes) 000032[0] 1788823347.341531 punch from 0x91a17c to 0x920159 (0x5fde bytes) 000035[0] 1788823347.396259 punch from 0x59fbbe to 0x5ac52c (0xc96f bytes) 000040[0] 1788823347.437632 punch from 0x2fb81e to 0x30a915 (0xf0f8 bytes) 000041[1] 1788823347.442554 trunc from 0x9504d5 to 0x015097 (0x93b43d bytes) truncating to largest ever: 0x99fbbc 000042[0] 1788823347.493217 trunc from 0x015098 to 0x99fbbb (0x98ab24 bytes) 000044[0] 1788823347.544867 trunc from 0x99fbbc to 0x53b190 (0x464a2b bytes) 000048[1] 1788823347.658104 trunc from 0x53b191 to 0x975884 (0x43a6f4 bytes) 000049[1] 1788823347.712470 punch from 0x904b80 to 0x913575 (0xe9f6 bytes) 000050[1] 1788823347.722547 mapwrite 0x8b8000 thru 0x8bd444 (0x5445 bytes) 000051[1] 1788823347.805219 trunc from 0x975885 to 0x08a84b (0x8eb039 bytes) 000055[1] 1788823347.862193 mapwrite 0x827000 thru 0x832fa5 (0xbfa6 bytes) 000057[1] 1788823347.965488 mapwrite 0x930000 thru 0x93eab4 (0xeab5 bytes) 000058[1] 1788823348.061918 punch from 0x2fc2c3 to 0x300a69 (0x47a7 bytes) 000061[1] 1788823348.109819 write 0x59e000 thru 0x5a2fe4 (0x4fe5 bytes) 000062[1] 1788823348.129900 read 0x2d5000 thru 0x2d73d0 (0x23d1 bytes) 000063[0] 1788823348.153718 mapwrite 0x10e000 thru 0x11b779 (0xd77a bytes) 000065[0] 1788823348.305404 punch from 0x907dca to 0x90d048 (0x527f bytes) 000066[0] 1788823348.324515 write 0x65a000 thru 0x666c9f (0xcca0 bytes) 000070[1] 1788823348.380191 trunc from 0x93eab5 to 0x85c577 (0xe253d bytes) 000073[1] 1788823348.454574 mapwrite 0x54f000 thru 0x5590fa (0xa0fb bytes) 000078[0] 1788823348.564846 trunc from 0x85c578 to 0x287006 (0x5d5571 bytes) 000080[1] 1788823348.643880 trunc from 0x287007 to 0x2c8b04 (0x41afe bytes) 000081[0] 1788823348.696900 mapread 0x19b000 thru 0x19e324 (0x3325 bytes) 000086[0] 1788823348.743354 punch from 0x012ed5 to 0x0218be (0xe9ea bytes) 000088[0] 1788823348.766526 punch from 0x0d18ca to 0x0d8533 (0x6c6a bytes) 000089[0] 1788823348.795137 trunc from 0x2c8b05 to 0x68fc9c (0x3c7198 bytes) 000091[1] 1788823348.861767 write 0x31b000 thru 0x323647 (0x8648 bytes) 000092[1] 1788823348.881087 mapread 0x073000 thru 0x080118 (0xd119 bytes) 000094[0] 1788823348.910349 read 0x452000 thru 0x458bda (0x6bdb bytes) 000096[1] 1788823348.947840 mapwrite 0x348000 thru 0x34d4b8 (0x54b9 bytes) 000097[1] 1788823349.076504 read 0x350000 thru 0x35f8ee (0xf8ef bytes) 000098[0] 1788823349.087176 write 0x23d000 thru 0x241bdb (0x4bdc bytes) 000100[1] 1788823349.165615 punch from 0x110c00 to 0x1111e1 (0x5e2 bytes) 000102[1] 1788823349.263284 read 0x584000 thru 0x5928cf (0xe8d0 bytes) 000103[1] 1788823349.270459 mapread 0x61e000 thru 0x623db1 (0x5db2 bytes) 000104[1] 1788823349.287529 write 0x6dc000 thru 0x6eb3a2 (0xf3a3 bytes) 000106[0] 1788823349.321899 punch from 0x66951c to 0x677032 (0xdb17 bytes) 000107[1] 1788823349.379316 write 0x84c000 thru 0x84d1fb (0x11fc bytes) 000109[0] 1788823349.394073 write 0x474000 thru 0x47e8fd (0xa8fe bytes) 000110[0] 1788823349.430589 punch from 0x733854 to 0x73aa68 (0x7215 bytes) 000112[0] 1788823349.435805 read 0x607000 thru 0x610159 (0x915a bytes) 000114[0] 1788823349.478255 punch from 0x0ba1e5 to 0x0c36ee (0x950a bytes) 000116[1] 1788823349.484152 read 0x56e000 thru 0x57634b (0x834c bytes) 000118[1] 1788823349.544238 trunc from 0x84d1fc to 0x19476c (0x6b8a8f bytes) 000119[1] 1788823349.646744 mapread 0x121000 thru 0x12a430 (0x9431 bytes) 000121[1] 1788823349.698031 mapwrite 0x842000 thru 0x84c853 (0xa854 bytes) 000122[0] 1788823349.894205 write 0x202000 thru 0x20d0e3 (0xb0e4 bytes) 000123[1] 1788823349.939698 punch from 0x2f7f21 to 0x3021a2 (0xa282 bytes) 000124[0] 1788823350.050597 mapread 0x304000 thru 0x30c33e (0x833f bytes) 000125[1] 1788823350.160643 read 0x718000 thru 0x727850 (0xf851 bytes) 000126[1] 1788823350.177569 read 0x425000 thru 0x433117 (0xe118 bytes) 000127[1] 1788823350.192301 punch from 0x0e3a6b to 0x0e744a (0x39e0 bytes) 000128[1] 1788823350.235928 read 0x0c2000 thru 0x0d1372 (0xf373 bytes) 000129[1] 1788823350.260917 write 0x7ce000 thru 0x7cf277 (0x1278 bytes) 000131[0] 1788823350.294252 punch from 0x560d16 to 0x567a2f (0x6d1a bytes) 000132[1] 1788823350.355096 mapwrite 0x04d000 thru 0x05362a (0x662b bytes) 000135[1] 1788823350.564859 punch from 0x746d4a to 0x754433 (0xd6ea bytes) 000136[1] 1788823350.577815 mapwrite 0x8da000 thru 0x8e03ce (0x63cf bytes) 000137[1] 1788823350.778155 mapread 0x365000 thru 0x3725c6 (0xd5c7 bytes) 000138[0] 1788823350.824070 mapwrite 0x24b000 thru 0x24ee7d (0x3e7e bytes) 000139[1] 1788823350.909878 punch from 0x1c580f to 0x1c6019 (0x80b bytes) 000141[0] 1788823350.954662 mapwrite 0x44a000 thru 0x44b98a (0x198b bytes) 000142[0] 1788823351.046140 write 0x078000 thru 0x079337 (0x1338 bytes) 000143[1] 1788823351.075628 write 0x4ee000 thru 0x4fbfb2 (0xdfb3 bytes) 000145[0] 1788823351.148329 mapwrite 0x548000 thru 0x54ccbe (0x4cbf bytes) 000146[0] 1788823351.427543 read 0x38c000 thru 0x38e0de (0x20df bytes) 000147[0] 1788823351.456839 trunc from 0x8e03cf to 0x7511ab (0x18f223 bytes) 000148[0] 1788823351.500269 mapread 0x4f0000 thru 0x4f9f66 (0x9f67 bytes) 000150[1] 1788823351.529740 mapwrite 0x22c000 thru 0x239dd3 (0xddd4 bytes) 000153[0] 1788823351.748144 trunc from 0x7511ac to 0x7a5128 (0x53f7d bytes) 000154[0] 1788823351.830104 mapwrite 0x382000 thru 0x387b31 (0x5b32 bytes) 000155[0] 1788823352.019244 trunc from 0x7a5129 to 0x892a6e (0xed946 bytes) 000157[1] 1788823352.143302 mapread 0x61a000 thru 0x61d033 (0x3034 bytes) 000159[1] 1788823352.275967 punch from 0x2385d5 to 0x23b722 (0x314e bytes) 000161[1] 1788823352.416376 punch from 0x62af5b to 0x63a9c2 (0xfa68 bytes) 000162[1] 1788823352.425056 mapwrite 0x3a0000 thru 0x3af0a3 (0xf0a4 bytes) 000163[1] 1788823352.652994 mapread 0x840000 thru 0x844b0e (0x4b0f bytes) 000164[0] 1788823352.723746 punch from 0x08fb86 to 0x0903bb (0x836 bytes) 000165[1] 1788823352.805523 mapread 0x091000 thru 0x097629 (0x662a bytes) 000166[0] 1788823352.885196 write 0x2bb000 thru 0x2c8832 (0xd833 bytes) 000169[1] 1788823352.938073 punch from 0x04e3a5 to 0x05415c (0x5db8 bytes) 000171[0] 1788823352.994105 punch from 0x4a4a61 to 0x4b002c (0xb5cc bytes) 000172[1] 1788823353.050313 write 0x0fd000 thru 0x1084be (0xb4bf bytes) 000175[0] 1788823353.097716 mapread 0x319000 thru 0x31c0a9 (0x30aa bytes) 000176[1] 1788823353.186107 mapread 0x695000 thru 0x69b6a8 (0x66a9 bytes) 000178[1] 1788823353.238178 write 0x6fe000 thru 0x70afe5 (0xcfe6 bytes) 000179[1] 1788823353.278604 read 0x270000 thru 0x27fe08 (0xfe09 bytes) 000180[1] 1788823353.348188 mapwrite 0x5d5000 thru 0x5e4c9c (0xfc9d bytes) 000181[0] 1788823353.558125 write 0x8fb000 thru 0x903306 (0x8307 bytes) 000182[1] 1788823353.611031 mapwrite 0x3f0000 thru 0x3fa9b5 (0xa9b6 bytes) 000183[1] 1788823353.784287 trunc from 0x903307 to 0x8f45d5 (0xed31 bytes) 000185[0] 1788823353.909548 punch from 0x7b71d9 to 0x7c53e7 (0xe20f bytes) 000186[0] 1788823353.958292 mapread 0x333000 thru 0x34243f (0xf440 bytes) 000187[1] 1788823354.000676 trunc from 0x8f45d6 to 0x54aa62 (0x3a9b73 bytes) 000189[1] 1788823354.091676 write 0x43a000 thru 0x43fa62 (0x5a63 bytes) 000190[1] 1788823354.136130 mapread 0x25c000 thru 0x25dd32 (0x1d33 bytes) 000191[0] 1788823354.174163 mapread 0x162000 thru 0x1666a8 (0x46a9 bytes) 000193[1] 1788823354.248575 mapwrite 0x2e0000 thru 0x2e4b8a (0x4b8b bytes) 000195[1] 1788823354.354811 trunc from 0x54aa63 to 0x44f3eb (0xfb677 bytes) 000197[1] 1788823354.385412 punch from 0x33e9d8 to 0x34bfd6 (0xd5ff bytes) 000198[0] 1788823354.394003 write 0x70f000 thru 0x71b03a (0xc03b bytes) 000199[1] 1788823354.417175 mapread 0x352000 thru 0x36183c (0xf83d bytes) 000200[0] 1788823354.479092 mapwrite 0x8d9000 thru 0x8d9394 (0x395 bytes) 000203[0] 1788823354.701509 trunc from 0x8d9395 to 0x5e2489 (0x2f6f0b bytes) 000204[0] 1788823354.762959 trunc from 0x5e248a to 0x3e56ab (0x1fcdde bytes) 000205[0] 1788823354.841734 trunc from 0x3e56ac to 0x09e1c1 (0x3474ea bytes) 000207[1] 1788823354.967149 mapread 0x09c000 thru 0x09da1c (0x1a1d bytes) 000211[1] 1788823355.075662 punch from 0x092ee2 to 0x095541 (0x2660 bytes) 000212[0] 1788823355.149083 punch from 0x08db26 to 0x09553d (0x7a18 bytes) 000213[1] 1788823355.215043 write 0x007000 thru 0x007705 (0x706 bytes) 000214[1] 1788823355.293969 trunc from 0x09e1c2 to 0x5557f4 (0x4b7633 bytes) 000216[1] 1788823355.454047 write 0x51d000 thru 0x52b47f (0xe480 bytes) 000217[0] 1788823355.503928 write 0x49b000 thru 0x4a47c4 (0x97c5 bytes) 000218[0] 1788823355.558009 mapwrite 0x0ff000 thru 0x10c88b (0xd88c bytes) 000219[0] 1788823355.806478 read 0x3a3000 thru 0x3a5126 (0x2127 bytes) 000220[1] 1788823355.874701 trunc from 0x5557f5 to 0x7d7891 (0x28209d bytes) 000221[1] 1788823355.937590 write 0x420000 thru 0x428a0d (0x8a0e bytes) 000222[1] 1788823355.994454 mapwrite 0x0a4000 thru 0x0a6289 (0x228a bytes) 000223[0] 1788823356.199982 punch from 0x7a886f to 0x7ace0a (0x459c bytes) 000224[0] 1788823356.251315 read 0x5c2000 thru 0x5c690d (0x490e bytes) 000226[1] 1788823356.265221 punch from 0x6e5f25 to 0x6e676e (0x84a bytes) 000227[1] 1788823356.320603 trunc from 0x7d7892 to 0x1418f4 (0x695f9d bytes) 000228[1] 1788823356.405517 read 0x0f5000 thru 0x0f7dca (0x2dcb bytes) 000229[0] 1788823356.580216 write 0x7fb000 thru 0x7ff706 (0x4707 bytes) 000231[1] 1788823356.718174 punch from 0x09eadf to 0x0a4fec (0x650e bytes) 000232[0] 1788823356.797266 mapread 0x228000 thru 0x22e8d3 (0x68d4 bytes) 000233[0] 1788823356.894299 punch from 0x473f57 to 0x47c4aa (0x8554 bytes) 000234[0] 1788823357.002991 punch from 0x6d2a52 to 0x6dec5a (0xc209 bytes) 000235[0] 1788823357.017202 mapwrite 0x978000 thru 0x97be66 (0x3e67 bytes) 000236[1] 1788823357.382320 punch from 0x164738 to 0x16eced (0xa5b6 bytes) 000237[0] 1788823357.500177 trunc from 0x97be67 to 0x41d110 (0x55ed56 bytes) 000238[1] 1788823357.589116 read 0x2f1000 thru 0x2fe0b2 (0xd0b3 bytes) 000241[0] 1788823357.655269 write 0x41e000 thru 0x424b9f (0x6ba0 bytes) 000242[1] 1788823357.688845 trunc from 0x424ba0 to 0x0faa53 (0x32a14c bytes) 000243[1] 1788823357.876662 mapread 0x0d7000 thru 0x0e6953 (0xf954 bytes) 000244[1] 1788823358.053542 write 0x4e5000 thru 0x4eeede (0x9edf bytes) 000249[0] 1788823358.185944 read 0x1b0000 thru 0x1bd31b (0xd31c bytes) 000250[1] 1788823358.303647 read 0x14b000 thru 0x155c5e (0xac5f bytes) 000251[1] 1788823358.351945 read 0x0dc000 thru 0x0e7b97 (0xbb98 bytes) 000252[0] 1788823358.386105 write 0x3c8000 thru 0x3d12a5 (0x92a6 bytes) 000253[1] 1788823358.499391 trunc from 0x4eeedf to 0x59e6bb (0xaf7dd bytes) 000254[1] 1788823358.651022 mapwrite 0x74f000 thru 0x75e3f8 (0xf3f9 bytes) 000256[0] 1788823359.164859 mapwrite 0x465000 thru 0x46b5c5 (0x65c6 bytes) 000257[0] 1788823359.471396 punch from 0x4fc541 to 0x4feb72 (0x2632 bytes) 000258[0] 1788823359.538631 write 0x0a8000 thru 0x0b0b7a (0x8b7b bytes) 000259[0] 1788823359.557940 trunc from 0x75e3f9 to 0x35ce53 (0x4015a5 bytes) 000260[0] 1788823359.651848 read 0x14c000 thru 0x14caa4 (0xaa5 bytes) 000262[1] 1788823359.867881 punch from 0x044ad4 to 0x0547e0 (0xfd0d bytes) 000263[1] 1788823359.969499 write 0x6f7000 thru 0x6f7aa5 (0xaa6 bytes) 000265[0] 1788823360.054024 read 0x6ee000 thru 0x6f33a4 (0x53a5 bytes) 000267[0] 1788823360.115673 write 0x295000 thru 0x29b79e (0x679f bytes) 000268[1] 1788823360.152051 mapread 0x224000 thru 0x225260 (0x1261 bytes) 000270[0] 1788823360.245081 mapwrite 0x5b9000 thru 0x5c5ec3 (0xcec4 bytes) 000271[1] 1788823360.452501 write 0x105000 thru 0x10ed64 (0x9d65 bytes) 000272[1] 1788823360.472187 punch from 0x31a22f to 0x325874 (0xb646 bytes) 000273[0] 1788823360.537546 read 0x3a9000 thru 0x3ac077 (0x3078 bytes) 000275[1] 1788823360.600938 trunc from 0x6f7aa6 to 0x2301d4 (0x4c78d1 bytes) 000277[1] 1788823360.739971 mapread 0x1b4000 thru 0x1c2560 (0xe561 bytes) 000278[0] 1788823360.847615 punch from 0x052afc to 0x059310 (0x6815 bytes) 000279[1] 1788823360.941302 write 0x45c000 thru 0x45d9dd (0x19de bytes) 000281[0] 1788823361.038877 mapwrite 0x9aa000 thru 0x9b9838 (0xf839 bytes) 000282[1] 1788823361.353008 write 0x42b000 thru 0x4362ba (0xb2bb bytes) 000283[1] 1788823361.415267 read 0x0fb000 thru 0x1032f1 (0x82f2 bytes) 000285[0] 1788823361.481785 write 0x7ad000 thru 0x7b5e82 (0x8e83 bytes) 000286[0] 1788823361.574822 mapread 0x8b2000 thru 0x8bcce1 (0xace2 bytes) 000287[0] 1788823361.604535 trunc from 0x9b9839 to 0x370699 (0x64919f bytes) 000288[1] 1788823361.672170 punch from 0x1bea6f to 0x1c32d5 (0x4867 bytes) 000289[1] 1788823361.731225 mapwrite 0x21f000 thru 0x22b733 (0xc734 bytes) 000290[0] 1788823361.948263 read 0x26f000 thru 0x271cac (0x2cad bytes) 000291[1] 1788823362.105269 write 0x8a3000 thru 0x8b07f3 (0xd7f4 bytes) 000292[0] 1788823362.246497 read 0x5ee000 thru 0x5eff88 (0x1f89 bytes) 000297[0] 1788823362.365613 read 0x0d2000 thru 0x0de37a (0xc37b bytes) 000298[1] 1788823362.379489 punch from 0x12c3c0 to 0x133aca (0x770b bytes) 000299[1] 1788823362.397620 trunc from 0x8b07f4 to 0x620b4c (0x28fca7 bytes) 000300[0] 1788823362.433056 write 0x4bd000 thru 0x4bd82c (0x82d bytes) 000301[0] 1788823362.464174 write 0x793000 thru 0x79c616 (0x9617 bytes) 000302[1] 1788823362.468476 trunc from 0x79c617 to 0x227c83 (0x574993 bytes) 000305[0] 1788823362.569113 trunc from 0x227c84 to 0x608b85 (0x3e0f02 bytes) 000307[0] 1788823362.628839 read 0x45b000 thru 0x461714 (0x6715 bytes) 000308[0] 1788823362.649097 mapread 0x446000 thru 0x45242a (0xc42b bytes) 000309[0] 1788823362.683701 mapwrite 0x576000 thru 0x57d17c (0x717d bytes) 000312[0] 1788823362.860411 mapwrite 0x2e2000 thru 0x2e293d (0x93e bytes) 000313[1] 1788823362.962903 read 0x0cf000 thru 0x0dc46a (0xd46b bytes) truncating to largest ever: 0x9d2dd5 000314[1] 1788823363.000168 trunc from 0x608b86 to 0x9d2dd4 (0x3ca24f bytes) 000317[1] 1788823363.076096 trunc from 0x9d2dd5 to 0x7caba3 (0x208231 bytes) 000318[1] 1788823363.121290 trunc from 0x7caba4 to 0x4cad3f (0x2ffe64 bytes) 000320[0] 1788823363.186238 trunc from 0x4cad40 to 0x124905 (0x3a643a bytes) 000321[1] 1788823363.248118 read 0x07d000 thru 0x0897cf (0xc7d0 bytes) 000324[0] 1788823363.340991 punch from 0x0abe5d to 0x0bb28c (0xf430 bytes) 000326[0] 1788823363.508849 write 0x1e9000 thru 0x1ee240 (0x5241 bytes) 000327[0] 1788823363.516193 punch from 0x1ca569 to 0x1d5ccb (0xb763 bytes) 000329[0] 1788823363.557355 punch from 0x0ff5af to 0x10da78 (0xe4ca bytes) 000333[1] 1788823363.615642 read 0x156000 thru 0x15d309 (0x730a bytes) 000334[1] 1788823363.812715 read 0x099000 thru 0x09b002 (0x2003 bytes) 000335[0] 1788823363.813124 write 0x286000 thru 0x294ed3 (0xeed4 bytes) 000336[0] 1788823363.879476 mapread 0x1da000 thru 0x1dde15 (0x3e16 bytes) 000339[0] 1788823363.940407 write 0x835000 thru 0x844095 (0xf096 bytes) 000341[1] 1788823363.992600 read 0x695000 thru 0x6a2056 (0xd057 bytes) 000342[1] 1788823364.090276 punch from 0x643c93 to 0x64f0e2 (0xb450 bytes) 000343[0] 1788823364.151037 punch from 0x5bc581 to 0x5c13d7 (0x4e57 bytes) 000345[1] 1788823364.348183 write 0x2a1000 thru 0x2b0513 (0xf514 bytes) 000346[1] 1788823364.397959 trunc from 0x844096 to 0x589b22 (0x2ba573 bytes) 000347[1] 1788823364.512184 trunc from 0x589b23 to 0x4ea1e7 (0x9f93b bytes) 000349[1] 1788823364.635464 read 0x40c000 thru 0x4172a8 (0xb2a9 bytes) 000350[0] 1788823364.737510 trunc from 0x4ea1e8 to 0x098518 (0x451ccf bytes) 000351[0] 1788823364.899932 punch from 0x0939b9 to 0x098518 (0x4b60 bytes) 000352[0] 1788823364.936982 punch from 0x011d66 to 0x0149d2 (0x2c6d bytes) 000355[1] 1788823365.001969 write 0x958000 thru 0x95d64b (0x564c bytes) 000359[1] 1788823365.099275 mapread 0x800000 thru 0x807a10 (0x7a11 bytes) 000360[1] 1788823365.148998 write 0x91a000 thru 0x91f853 (0x5854 bytes) 000361[0] 1788823365.176339 mapread 0x917000 thru 0x922441 (0xb442 bytes) 000362[1] 1788823365.257306 mapread 0x17e000 thru 0x189dbe (0xbdbf bytes) 000363[1] 1788823365.294741 write 0x0ef000 thru 0x0f9565 (0xa566 bytes) 000365[0] 1788823365.352293 write 0x9e0000 thru 0x9e8932 (0x8933 bytes) 000366[1] 1788823365.394584 mapwrite 0x685000 thru 0x6923d9 (0xd3da bytes) 000369[1] 1788823365.930305 punch from 0x2b25d9 to 0x2c19c6 (0xf3ee bytes) 000371[1] 1788823365.942099 write 0x04c000 thru 0x04df31 (0x1f32 bytes) 000372[0] 1788823366.006962 read 0x203000 thru 0x205709 (0x270a bytes) 000374[1] 1788823366.055182 read 0x0be000 thru 0x0ca0d4 (0xc0d5 bytes) 000376[0] 1788823366.065954 mapwrite 0x183000 thru 0x187c39 (0x4c3a bytes) 000377[0] 1788823366.141381 mapwrite 0x2db000 thru 0x2db1dc (0x1dd bytes) 000379[1] 1788823366.196875 read 0x5f6000 thru 0x603498 (0xd499 bytes) 000381[1] 1788823366.228022 write 0x797000 thru 0x79fac3 (0x8ac4 bytes) 000383[1] 1788823366.252808 write 0x3b1000 thru 0x3b12e0 (0x2e1 bytes) 000388[0] 1788823366.375019 mapread 0x5dd000 thru 0x5deaca (0x1acb bytes) 000389[1] 1788823366.486690 mapwrite 0x3a0000 thru 0x3a4a72 (0x4a73 bytes) 000391[1] 1788823366.793943 trunc from 0x9e8933 to 0x7451fe (0x2a3734 bytes) 000395[1] 1788823366.926251 read 0x5d4000 thru 0x5dd21f (0x9220 bytes) 000396[1] 1788823366.973148 mapwrite 0x1df000 thru 0x1eaf05 (0xbf06 bytes) 000398[0] 1788823367.313055 trunc from 0x7451ff to 0x4ee7c0 (0x256a3e bytes) 000400[0] 1788823367.368871 mapwrite 0x470000 thru 0x472261 (0x2262 bytes) 000401[1] 1788823367.510154 punch from 0x3118b3 to 0x312b2b (0x1279 bytes) 000402[0] 1788823367.598796 read 0x3c7000 thru 0x3cbef5 (0x4ef6 bytes) 000403[1] 1788823367.691762 mapread 0x2a1000 thru 0x2ac190 (0xb191 bytes) 000404[1] 1788823367.731572 mapread 0x2c8000 thru 0x2d76ff (0xf700 bytes) 000410[0] 1788823367.780131 write 0x993000 thru 0x99db33 (0xab34 bytes) 000412[1] 1788823367.986773 write 0x86a000 thru 0x86e92f (0x4930 bytes) 000414[1] 1788823368.136217 trunc from 0x99db34 to 0x571bda (0x42bf59 bytes) 000415[0] 1788823368.160004 mapread 0x317000 thru 0x32316b (0xc16c bytes) 000417[0] 1788823368.204989 mapwrite 0x763000 thru 0x770feb (0xdfec bytes) 000418[0] 1788823368.440677 punch from 0x21932a to 0x226532 (0xd209 bytes) 000419[1] 1788823368.474732 punch from 0x6e9206 to 0x6f55c7 (0xc3c2 bytes) 000420[1] 1788823368.536368 mapwrite 0x246000 thru 0x24fb4c (0x9b4d bytes) 000421[0] 1788823368.716070 mapwrite 0x29f000 thru 0x2a8dbc (0x9dbd bytes) 000423[0] 1788823368.908034 write 0x311000 thru 0x3153c3 (0x43c4 bytes) 000426[1] 1788823368.963483 punch from 0x0b5970 to 0x0bea0b (0x909c bytes) 000427[0] 1788823368.993729 mapwrite 0x072000 thru 0x07daf1 (0xbaf2 bytes) 000431[1] 1788823369.273303 punch from 0x51fa83 to 0x526ade (0x705c bytes) 000432[1] 1788823369.325143 trunc from 0x770fec to 0x1931a1 (0x5dde4a bytes) 000433[0] 1788823369.362484 mapwrite 0x617000 thru 0x61e038 (0x7039 bytes) 000434[0] 1788823369.748344 read 0x58e000 thru 0x59c420 (0xe421 bytes) 000435[0] 1788823369.792116 write 0x00b000 thru 0x01af16 (0xff17 bytes) 000436[0] 1788823369.805184 trunc from 0x61e039 to 0x40ac7b (0x2133bd bytes) 000437[1] 1788823369.826909 mapread 0x0d5000 thru 0x0d9840 (0x4841 bytes) 000438[0] 1788823369.974331 read 0x0b4000 thru 0x0c0bcc (0xcbcd bytes) 000439[1] 1788823369.988827 mapwrite 0x5a6000 thru 0x5b09fc (0xa9fd bytes) 000441[1] 1788823370.239028 write 0x2ef000 thru 0x2f1439 (0x243a bytes) 000443[1] 1788823370.357382 mapread 0x204000 thru 0x20fe2d (0xbe2e bytes) 000444[0] 1788823370.534524 mapwrite 0x892000 thru 0x895cf0 (0x3cf1 bytes) 000446[1] 1788823370.823645 mapwrite 0x001000 thru 0x00c922 (0xb923 bytes) 000448[1] 1788823371.224750 mapwrite 0x1dd000 thru 0x1e12aa (0x42ab bytes) 000450[1] 1788823371.464997 mapwrite 0x5dc000 thru 0x5e4b75 (0x8b76 bytes) 000451[0] 1788823371.695167 punch from 0x57c76f to 0x5897ed (0xd07f bytes) 000453[1] 1788823371.779722 punch from 0x235770 to 0x239e8d (0x471e bytes) 000456[1] 1788823371.864353 trunc from 0x895cf1 to 0x36afe4 (0x52ad0c bytes) 000457[0] 1788823371.932165 trunc from 0x36afe5 to 0x478140 (0x10d15c bytes) 000461[1] 1788823372.171950 read 0x29b000 thru 0x2a63a7 (0xb3a8 bytes) 000463[1] 1788823372.241684 trunc from 0x478141 to 0x583a54 (0x10b914 bytes) 000464[1] 1788823372.321657 punch from 0x369060 to 0x371952 (0x88f3 bytes) 000465[0] 1788823372.333303 mapwrite 0x502000 thru 0x507935 (0x5936 bytes) 000467[1] 1788823372.626338 mapwrite 0x0c1000 thru 0x0cb980 (0xa981 bytes) 000468[0] 1788823372.942206 read 0x3ab000 thru 0x3b7e6f (0xce70 bytes) 000469[1] 1788823372.984106 trunc from 0x583a55 to 0x6d08b7 (0x14ce63 bytes) 000473[1] 1788823373.044753 trunc from 0x6d08b8 to 0x6adf55 (0x22962 bytes) 000476[1] 1788823373.121347 read 0x11d000 thru 0x12aa94 (0xda95 bytes) 000477[1] 1788823373.162954 write 0x2e6000 thru 0x2f2c9b (0xcc9c bytes) 000480[1] 1788823373.238274 read 0x42f000 thru 0x4315a8 (0x25a9 bytes) 000481[1] 1788823373.278236 read 0x2a2000 thru 0x2a3dae (0x1daf bytes) 000482[0] 1788823373.305774 read 0x0b1000 thru 0x0b44a0 (0x34a1 bytes) 000483[1] 1788823373.362921 write 0x5f1000 thru 0x5f1f5d (0xf5e bytes) 000484[1] 1788823373.388827 punch from 0x1a3e9c to 0x1acf2c (0x9091 bytes) 000485[1] 1788823373.399638 read 0x23e000 thru 0x2406a7 (0x26a8 bytes) 000491[0] 1788823373.438392 mapwrite 0x90f000 thru 0x915e7c (0x6e7d bytes) 000492[1] 1788823373.640951 trunc from 0x915e7d to 0x2657d0 (0x6b06ac bytes) 000493[1] 1788823373.668822 mapread 0x11a000 thru 0x122d3d (0x8d3e bytes) 000494[0] 1788823373.711313 trunc from 0x2657d1 to 0x071ae0 (0x1f3cf0 bytes) 000499[1] 1788823373.881319 mapwrite 0x449000 thru 0x456fae (0xdfaf bytes) All operations completed A-OK! PASS 16a (66s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 19:23:03 (1788823383) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.302329 s, 27.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.186966 s, 44.9 MB/s Chance of close/open is 1 in 50 Seed set to 3386 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004[0] 1788823387.837053 mapwrite 0x36e19a thru 0x36e8fd (0x764 bytes) truncating to largest ever: 0x4224d3 000005[1] 1788823388.250818 trunc from 0x36e8fe to 0x4224d2 (0xb3bd5 bytes) truncating to largest ever: 0x556e5e 000006[0] 1788823388.392737 trunc from 0x4224d3 to 0x556e5d (0x13498b bytes) truncating to largest ever: 0x7d319f 000008[0] 1788823388.738712 trunc from 0x556e5e to 0x7d319e (0x27c341 bytes) 000009[1] 1788823388.799141 punch from 0x6eca21 to 0x6ee407 (0x19e7 bytes) 000010[0] 1788823388.890243 read 0x46e6ad thru 0x46e6b9 (0xd bytes) truncating to largest ever: 0x7dc5d0 000012[0] 1788823388.923084 trunc from 0x7d319f to 0x7dc5cf (0x9431 bytes) 000013[0] 1788823389.010619 write 0x428c09 thru 0x4294a0 (0x898 bytes) truncating to largest ever: 0x8db841 000014[0] 1788823389.120533 trunc from 0x7dc5d0 to 0x8db840 (0xff271 bytes) 000017[0] 1788823389.192459 punch from 0x0c6268 to 0x0c6712 (0x4ab bytes) 000019[1] 1788823389.296181 mapwrite 0x17601b thru 0x176336 (0x31c bytes) 000020[0] 1788823389.581562 trunc from 0x8db841 to 0x2e1563 (0x5fa2dd bytes) 000023[0] 1788823389.718548 punch from 0x078f76 to 0x0793f0 (0x47b bytes) 000024[1] 1788823389.743519 mapwrite 0x2bad62 thru 0x2bb46b (0x70a bytes) 000025[0] 1788823389.835946 mapread 0x2cfac4 thru 0x2d0208 (0x745 bytes) 000027[0] 1788823389.868824 trunc from 0x2e1564 to 0x151918 (0x18fc4b bytes) truncating to largest ever: 0x98048d 000028[1] 1788823389.930897 trunc from 0x151919 to 0x98048c (0x82eb74 bytes) 000029[0] 1788823389.973238 mapread 0x7a0f6a thru 0x7a20a6 (0x113d bytes) 000030[0] 1788823390.025545 trunc from 0x98048d to 0x87d34e (0x10313e bytes) truncating to largest ever: 0x9ffcca 000032[0] 1788823390.063065 trunc from 0x87d34f to 0x9ffcc9 (0x18297b bytes) 000034[1] 1788823390.115092 read 0x8fc6b6 thru 0x8fe4e2 (0x1e2d bytes) 000035[1] 1788823390.141291 read 0x0d5ebe thru 0x0d6eb3 (0xff6 bytes) 000036[1] 1788823390.164900 mapwrite 0x9b8818 thru 0x9b89df (0x1c8 bytes) 000037[1] 1788823390.278126 mapwrite 0x44640d thru 0x446b99 (0x78d bytes) 000038[0] 1788823390.406289 read 0x2b6436 thru 0x2b799c (0x1567 bytes) 000039[0] 1788823390.463794 mapread 0x89771f thru 0x89787d (0x15f bytes) 000040[1] 1788823390.528987 read 0x062735 thru 0x062cd2 (0x59e bytes) 000043[1] 1788823390.540083 mapwrite 0x8a72b5 thru 0x8a8afe (0x184a bytes) 000046[1] 1788823390.677948 write 0x150d84 thru 0x151867 (0xae4 bytes) 000048[0] 1788823390.722890 trunc from 0x9ffcca to 0x27df2d (0x781d9c bytes) 000049[0] 1788823390.858984 punch from 0x27badb to 0x27cd17 (0x123d bytes) 000053[1] 1788823391.085115 mapwrite 0x066d3e thru 0x0673a9 (0x66c bytes) 000056[0] 1788823391.338822 read 0x18b270 thru 0x18b7b1 (0x542 bytes) 000057[0] 1788823391.542663 read 0x146709 thru 0x147000 (0x8f8 bytes) 000061[1] 1788823391.594738 trunc from 0x27df2e to 0x888825 (0x60a8f8 bytes) 000063[1] 1788823391.739870 mapwrite 0x8ab703 thru 0x8ac461 (0xd5f bytes) 000065[0] 1788823391.870554 punch from 0x4f0b60 to 0x4f19fa (0xe9b bytes) 000071[1] 1788823392.101944 read 0x06b01b thru 0x06b51a (0x500 bytes) 000072[0] 1788823392.128363 read 0x7d4c77 thru 0x7d5d28 (0x10b2 bytes) 000073[0] 1788823392.153040 write 0x46152c thru 0x462424 (0xef9 bytes) 000075[0] 1788823392.232043 mapread 0x42f4c0 thru 0x42ff2e (0xa6f bytes) 000076[1] 1788823392.282034 trunc from 0x8ac462 to 0x41d10a (0x48f357 bytes) 000077[1] 1788823392.342824 read 0x26adf8 thru 0x26b514 (0x71d bytes) 000079[1] 1788823392.390044 mapwrite 0x8fbd2a thru 0x8fcbf9 (0xed0 bytes) 000081[0] 1788823392.697038 mapread 0x82de80 thru 0x82f673 (0x17f4 bytes) 000082[0] 1788823392.822019 mapwrite 0x4724e9 thru 0x472eca (0x9e2 bytes) 000083[0] 1788823393.000777 read 0x61dc8b thru 0x61e3ee (0x764 bytes) 000084[0] 1788823393.066686 write 0x9ad471 thru 0x9af2ff (0x1e8f bytes) 000086[0] 1788823393.189635 write 0x16283e thru 0x162bdc (0x39f bytes) 000087[0] 1788823393.195506 trunc from 0x9af300 to 0x086bf3 (0x92870c bytes) 000093[0] 1788823393.408333 mapread 0x005c5d thru 0x005fd0 (0x374 bytes) 000094[1] 1788823393.429755 mapwrite 0x5dc723 thru 0x5dd827 (0x1105 bytes) 000095[0] 1788823393.511753 punch from 0x3f8a56 to 0x3f9d36 (0x12e1 bytes) 000096[0] 1788823393.530994 mapread 0x3977da thru 0x397d9b (0x5c2 bytes) 000097[1] 1788823393.566686 read 0x17bd44 thru 0x17c287 (0x544 bytes) 000098[0] 1788823393.626717 mapwrite 0x94571b thru 0x94625c (0xb42 bytes) 000103[1] 1788823393.830481 mapread 0x6bd53b thru 0x6bf167 (0x1c2d bytes) 000104[1] 1788823393.872681 punch from 0x6862df to 0x687863 (0x1585 bytes) 000105[1] 1788823393.903243 mapread 0x03bda1 thru 0x03c54f (0x7af bytes) 000110[0] 1788823393.989965 mapread 0x4b7a51 thru 0x4b8aff (0x10af bytes) 000111[0] 1788823394.052896 trunc from 0x94625d to 0x25e62a (0x6e7c32 bytes) 000112[0] 1788823394.198016 punch from 0x105cce to 0x1075c9 (0x18fc bytes) 000113[0] 1788823394.272899 mapwrite 0x01b6f9 thru 0x01c7d0 (0x10d8 bytes) 000115[1] 1788823394.408051 trunc from 0x25e62b to 0x06717b (0x1f74af bytes) 000116[1] 1788823394.480231 mapwrite 0x73eb7a thru 0x73f3ef (0x876 bytes) 000119[0] 1788823394.556014 punch from 0x388218 to 0x389ebf (0x1ca8 bytes) 000120[0] 1788823394.629507 punch from 0x48fc18 to 0x490da4 (0x118d bytes) 000121[1] 1788823394.691724 trunc from 0x73f3f0 to 0x43645a (0x308f95 bytes) 000123[1] 1788823394.814208 write 0x9a9860 thru 0x9aa718 (0xeb9 bytes) 000124[1] 1788823394.900920 trunc from 0x9aa719 to 0x4a3b3d (0x506bdb bytes) 000125[0] 1788823394.950287 write 0x3db316 thru 0x3dd22d (0x1f18 bytes) 000126[1] 1788823395.030733 trunc from 0x4a3b3e to 0x0a8119 (0x3fba24 bytes) 000127[1] 1788823395.137090 trunc from 0x0a811a to 0x0e41bd (0x3c0a4 bytes) 000128[1] 1788823395.185467 trunc from 0x0e41be to 0x0dcc08 (0x75b5 bytes) 000129[0] 1788823395.232798 trunc from 0x0dcc09 to 0x123690 (0x46a88 bytes) 000132[0] 1788823395.300528 read 0x01fa98 thru 0x01ff21 (0x48a bytes) 000134[0] 1788823395.361894 write 0x16f690 thru 0x16fef6 (0x867 bytes) 000135[1] 1788823395.389846 punch from 0x01894e to 0x0196de (0xd91 bytes) 000136[1] 1788823395.428778 read 0x0f958b thru 0x0f990f (0x385 bytes) 000137[0] 1788823395.498146 mapwrite 0x7b0528 thru 0x7b17c3 (0x129c bytes) 000139[0] 1788823395.749269 mapread 0x70757c thru 0x7076fd (0x182 bytes) 000140[1] 1788823395.757253 trunc from 0x7b17c4 to 0x235529 (0x57c29a bytes) 000141[0] 1788823395.797391 read 0x11d29f thru 0x11e649 (0x13ab bytes) 000143[0] 1788823395.865746 punch from 0x1c8980 to 0x1c9390 (0xa11 bytes) 000146[0] 1788823395.989912 read 0x129265 thru 0x129538 (0x2d4 bytes) 000149[0] 1788823396.422156 mapread 0x153b0d thru 0x1554e5 (0x19d9 bytes) 000150[0] 1788823396.491002 mapread 0x10581b thru 0x106173 (0x959 bytes) 000151[0] 1788823396.527306 punch from 0x1bdb07 to 0x1be249 (0x743 bytes) 000155[1] 1788823396.575371 read 0x0535d7 thru 0x053c6b (0x695 bytes) 000156[0] 1788823396.617267 trunc from 0x23552a to 0x389bdd (0x1546b4 bytes) 000157[1] 1788823396.710807 trunc from 0x389bde to 0x21833a (0x1718a3 bytes) 000158[1] 1788823396.825404 write 0x9680db thru 0x968b6b (0xa91 bytes) 000159[1] 1788823396.876236 trunc from 0x968b6c to 0x72db84 (0x23afe7 bytes) 000160[1] 1788823396.897284 punch from 0x53180b to 0x532430 (0xc26 bytes) 000162[0] 1788823396.938246 mapread 0x2bf40e thru 0x2c099d (0x1590 bytes) 000163[1] 1788823397.100625 read 0x0fcec8 thru 0x0fd95c (0xa95 bytes) 000164[0] 1788823397.152300 mapwrite 0x7e811d thru 0x7e9412 (0x12f6 bytes) 000166[0] 1788823397.741671 write 0x0f282a thru 0x0f404d (0x1824 bytes) 000170[0] 1788823397.934919 punch from 0x273bbe to 0x275b62 (0x1fa5 bytes) 000171[1] 1788823398.075987 punch from 0x288c67 to 0x28a490 (0x182a bytes) 000173[1] 1788823398.336335 trunc from 0x7e9413 to 0x30711a (0x4e22f8 bytes) 000175[1] 1788823398.453478 punch from 0x09e26f to 0x09eb76 (0x908 bytes) 000180[0] 1788823398.594975 punch from 0x0c8aaf to 0x0ca71a (0x1c6c bytes) 000181[1] 1788823398.661508 write 0x186efd thru 0x188b7d (0x1c81 bytes) 000183[0] 1788823398.708715 punch from 0x163683 to 0x165477 (0x1df5 bytes) 000186[1] 1788823398.844962 punch from 0x2a5da4 to 0x2a7803 (0x1a60 bytes) 000187[0] 1788823398.937915 punch from 0x2e02b5 to 0x2e058e (0x2da bytes) 000188[0] 1788823399.000771 read 0x0cbe13 thru 0x0cbf2e (0x11c bytes) 000189[1] 1788823399.204305 write 0x37cd92 thru 0x37d171 (0x3e0 bytes) 000190[1] 1788823399.623759 mapread 0x2d7c5c thru 0x2d939f (0x1744 bytes) 000192[0] 1788823399.892867 read 0x29b752 thru 0x29bc3f (0x4ee bytes) 000198[0] 1788823400.250376 read 0x04fe66 thru 0x0505dc (0x777 bytes) 000199[0] 1788823400.251233 mapread 0x33c29f thru 0x33ca8f (0x7f1 bytes) 000200[0] 1788823400.301788 mapwrite 0x73d4ff thru 0x73ecef (0x17f1 bytes) 000203[1] 1788823401.007791 mapread 0x033beb thru 0x035265 (0x167b bytes) 000205[1] 1788823401.311342 punch from 0x590c51 to 0x59133a (0x6ea bytes) 000206[1] 1788823401.436156 mapread 0x211e52 thru 0x21293c (0xaeb bytes) 000207[0] 1788823401.476457 mapread 0x592046 thru 0x592b65 (0xb20 bytes) 000210[0] 1788823401.587263 mapread 0x419df7 thru 0x41bb38 (0x1d42 bytes) 000211[0] 1788823401.608999 write 0x871bcf thru 0x871bd1 (0x3 bytes) 000212[0] 1788823401.651280 punch from 0x59c9e7 to 0x59e66b (0x1c85 bytes) 000213[0] 1788823401.703838 mapread 0x2839a2 thru 0x2841a7 (0x806 bytes) 000214[1] 1788823401.716651 mapread 0x5bdd15 thru 0x5bef3a (0x1226 bytes) 000215[1] 1788823401.786390 trunc from 0x871bd2 to 0x42645d (0x44b774 bytes) 000217[1] 1788823401.908268 read 0x0c4501 thru 0x0c46ab (0x1ab bytes) 000219[1] 1788823401.919584 write 0x6597ad thru 0x65a66c (0xec0 bytes) 000220[1] 1788823401.954213 punch from 0x0cb1df to 0x0cc8c5 (0x16e7 bytes) 000222[0] 1788823402.018653 punch from 0x5c1452 to 0x5c3235 (0x1de4 bytes) 000225[1] 1788823402.094961 punch from 0x3cd1b3 to 0x3cede7 (0x1c35 bytes) 000226[1] 1788823402.117957 mapwrite 0x380f2e thru 0x382e12 (0x1ee5 bytes) 000229[1] 1788823402.250284 punch from 0x5c2144 to 0x5c3af4 (0x19b1 bytes) 000231[0] 1788823402.343621 punch from 0x0f9441 to 0x0f9c30 (0x7f0 bytes) 000232[1] 1788823402.386284 mapread 0x4125c4 thru 0x41332e (0xd6b bytes) 000233[1] 1788823402.422110 trunc from 0x65a66d to 0x732416 (0xd7daa bytes) 000234[0] 1788823402.485944 write 0x8f2b18 thru 0x8f424a (0x1733 bytes) 000235[0] 1788823402.550888 write 0x831a22 thru 0x831cd7 (0x2b6 bytes) 000239[0] 1788823402.630120 mapwrite 0x82ab4a thru 0x82bff3 (0x14aa bytes) 000240[0] 1788823402.724578 write 0x5297cd thru 0x52a877 (0x10ab bytes) 000246[1] 1788823402.853329 mapwrite 0x9d6902 thru 0x9d6f90 (0x68f bytes) 000249[0] 1788823402.995777 mapwrite 0x5c1f3b thru 0x5c27f3 (0x8b9 bytes) 000250[0] 1788823403.092714 trunc from 0x9d6f91 to 0x3d28b2 (0x6046de bytes) 000254[1] 1788823403.163861 mapread 0x04d773 thru 0x04e0f4 (0x982 bytes) 000258[1] 1788823403.255298 punch from 0x37eea0 to 0x37f302 (0x463 bytes) 000259[1] 1788823403.296239 punch from 0x3c8f8f to 0x3ca522 (0x1594 bytes) 000260[1] 1788823403.321531 trunc from 0x3d28b3 to 0x6e016c (0x30d8ba bytes) 000261[0] 1788823403.389889 mapread 0x367ff8 thru 0x3694fa (0x1503 bytes) 000262[0] 1788823403.464341 punch from 0x646560 to 0x64833e (0x1ddf bytes) 000263[1] 1788823403.486648 mapread 0x2a5316 thru 0x2a71d0 (0x1ebb bytes) 000266[0] 1788823403.554477 mapwrite 0x4b2d3d thru 0x4b2f11 (0x1d5 bytes) 000267[0] 1788823403.644343 mapread 0x057bbf thru 0x059804 (0x1c46 bytes) 000268[1] 1788823403.675482 write 0x672b34 thru 0x672b7c (0x49 bytes) 000270[1] 1788823403.711389 mapwrite 0x92e6ec thru 0x9302a6 (0x1bbb bytes) 000272[0] 1788823403.952374 write 0x107987 thru 0x1088d7 (0xf51 bytes) 000274[0] 1788823404.015548 read 0x301523 thru 0x302c13 (0x16f1 bytes) 000275[0] 1788823404.036808 write 0x87abf3 thru 0x87b585 (0x993 bytes) 000276[1] 1788823404.062545 mapread 0x895411 thru 0x895a6d (0x65d bytes) 000277[0] 1788823404.101309 read 0x844e16 thru 0x84538c (0x577 bytes) 000278[0] 1788823404.109791 read 0x161ea3 thru 0x16395b (0x1ab9 bytes) 000280[0] 1788823404.129252 mapwrite 0x8bc886 thru 0x8bda4c (0x11c7 bytes) 000282[1] 1788823404.237171 trunc from 0x9302a7 to 0x19d1c9 (0x7930dd bytes) 000283[1] 1788823404.318893 trunc from 0x19d1ca to 0x25e639 (0xc1470 bytes) 000284[1] 1788823404.367722 write 0x682e3e thru 0x684a14 (0x1bd7 bytes) 000285[1] 1788823404.392059 write 0x95b9b2 thru 0x95d684 (0x1cd3 bytes) 000287[1] 1788823404.401241 write 0x2f63d3 thru 0x2f6f5b (0xb89 bytes) 000289[1] 1788823404.459723 mapread 0x368f92 thru 0x36aa24 (0x1a93 bytes) 000291[1] 1788823404.484174 mapwrite 0x8fed03 thru 0x900597 (0x1895 bytes) 000292[0] 1788823404.582198 trunc from 0x95d685 to 0x92456c (0x39118 bytes) 000293[0] 1788823404.635068 punch from 0x5914ce to 0x592125 (0xc58 bytes) 000294[0] 1788823404.666673 read 0x89b61e thru 0x89b69c (0x7f bytes) 000295[0] 1788823404.670388 mapread 0x2a2474 thru 0x2a2b61 (0x6ee bytes) 000297[1] 1788823404.674647 mapwrite 0x68f3e0 thru 0x69027e (0xe9f bytes) 000299[0] 1788823404.909789 trunc from 0x92456d to 0x4dfd9a (0x4447d2 bytes) 000300[0] 1788823404.946716 mapwrite 0x9e96e0 thru 0x9eae40 (0x1761 bytes) 000301[1] 1788823405.030358 trunc from 0x9eae41 to 0x3afa80 (0x63b3c0 bytes) 000304[1] 1788823405.113613 mapread 0x2cdc64 thru 0x2cf799 (0x1b36 bytes) 000305[1] 1788823405.148695 mapread 0x1645b5 thru 0x165b89 (0x15d5 bytes) 000308[0] 1788823405.297532 punch from 0x03c4d5 to 0x03e08c (0x1bb8 bytes) 000309[0] 1788823405.393311 trunc from 0x3afa81 to 0x908a40 (0x558fc0 bytes) 000312[0] 1788823405.562258 punch from 0x426666 to 0x4274eb (0xe86 bytes) 000313[1] 1788823405.626727 mapread 0x5c3a11 thru 0x5c3e6e (0x45e bytes) 000314[0] 1788823405.690929 mapread 0x4f5fd0 thru 0x4f6a9c (0xacd bytes) 000315[0] 1788823405.739514 punch from 0x106a03 to 0x10751c (0xb1a bytes) 000317[1] 1788823405.797147 mapwrite 0x861cb3 thru 0x8635cf (0x191d bytes) 000318[1] 1788823405.930474 trunc from 0x908a41 to 0x4665d5 (0x4a246b bytes) 000319[0] 1788823405.977240 trunc from 0x4665d6 to 0x8b5aa3 (0x44f4ce bytes) 000320[0] 1788823406.022108 read 0x5a1df1 thru 0x5a23da (0x5ea bytes) 000323[0] 1788823406.082330 mapread 0x38f0e0 thru 0x390abb (0x19dc bytes) 000324[0] 1788823406.114225 trunc from 0x8b5aa4 to 0x6b91ef (0x1fc8b4 bytes) 000325[0] 1788823406.146013 write 0x810d3f thru 0x812bb8 (0x1e7a bytes) 000326[1] 1788823406.148598 mapwrite 0x489852 thru 0x48a601 (0xdb0 bytes) 000327[1] 1788823406.274731 read 0x503805 thru 0x5053ee (0x1bea bytes) 000332[0] 1788823406.430915 read 0x2074e9 thru 0x208ebc (0x19d4 bytes) 000333[0] 1788823406.472246 mapwrite 0x8d1032 thru 0x8d14f7 (0x4c6 bytes) 000335[0] 1788823406.764639 trunc from 0x8d14f8 to 0x9295d0 (0x580d9 bytes) 000337[0] 1788823406.835408 write 0x1decd9 thru 0x1e0ad1 (0x1df9 bytes) 000338[1] 1788823406.870143 mapwrite 0x36b503 thru 0x36d214 (0x1d12 bytes) 000340[1] 1788823407.044336 write 0x8035d8 thru 0x803bf4 (0x61d bytes) 000341[1] 1788823407.125912 mapwrite 0x3aad15 thru 0x3abd20 (0x100c bytes) 000344[0] 1788823407.368182 mapread 0x521a7f thru 0x523856 (0x1dd8 bytes) 000345[1] 1788823407.475508 read 0x81cd9e thru 0x81dcc8 (0xf2b bytes) 000347[0] 1788823407.563235 mapread 0x507601 thru 0x5081e4 (0xbe4 bytes) 000349[1] 1788823407.668086 punch from 0x584eba to 0x585c4b (0xd92 bytes) 000357[1] 1788823407.900536 write 0x2be95d thru 0x2bf6db (0xd7f bytes) 000361[0] 1788823407.956258 mapread 0x78bc3b thru 0x78d0e8 (0x14ae bytes) 000362[1] 1788823408.004852 trunc from 0x9295d1 to 0x6e755d (0x242073 bytes) 000364[0] 1788823408.082002 trunc from 0x6e755e to 0x0ed0f7 (0x5fa466 bytes) 000365[1] 1788823408.156492 write 0x4129ca thru 0x41336d (0x9a4 bytes) 000367[1] 1788823408.203239 trunc from 0x41336e to 0x4ed123 (0xd9db6 bytes) 000368[0] 1788823408.288877 trunc from 0x4ed124 to 0x8d6979 (0x3e9856 bytes) 000370[1] 1788823408.484195 mapread 0x6aabe9 thru 0x6abd3b (0x1153 bytes) 000372[1] 1788823408.567595 read 0x17a480 thru 0x17bc25 (0x17a6 bytes) 000373[0] 1788823408.578783 write 0x3320c7 thru 0x333a80 (0x19ba bytes) 000375[0] 1788823408.627525 write 0x0b015d thru 0x0b1f52 (0x1df6 bytes) 000378[1] 1788823408.695049 mapwrite 0x22305c thru 0x22328e (0x233 bytes) 000379[1] 1788823408.837550 read 0x2fbefe thru 0x2fc538 (0x63b bytes) 000381[0] 1788823408.929016 punch from 0x774a68 to 0x775281 (0x81a bytes) 000382[1] 1788823409.002172 mapwrite 0x671da1 thru 0x67375a (0x19ba bytes) 000383[0] 1788823409.210601 write 0x2bbd6a thru 0x2bd15b (0x13f2 bytes) 000388[1] 1788823409.398676 mapwrite 0x27211e thru 0x273e02 (0x1ce5 bytes) 000392[1] 1788823409.726399 mapwrite 0x4f66f2 thru 0x4f6abf (0x3ce bytes) 000393[0] 1788823409.878861 read 0x12ae4c thru 0x12b0fe (0x2b3 bytes) 000395[1] 1788823409.964139 write 0x71e4a3 thru 0x71f4e0 (0x103e bytes) 000396[0] 1788823410.047986 trunc from 0x8d697a to 0x95f15e (0x887e5 bytes) 000397[0] 1788823410.153161 write 0x9dbf8e thru 0x9dda14 (0x1a87 bytes) 000398[1] 1788823410.154451 punch from 0x201a44 to 0x202bfc (0x11b9 bytes) 000399[1] 1788823410.253087 read 0x91f71c thru 0x9201cf (0xab4 bytes) 000400[1] 1788823410.287139 punch from 0x2b68a3 to 0x2b78ea (0x1048 bytes) 000401[0] 1788823410.338593 mapread 0x979157 thru 0x97ad6d (0x1c17 bytes) 000404[1] 1788823410.509083 trunc from 0x9dda15 to 0x9235e2 (0xba432 bytes) 000407[1] 1788823410.682482 read 0x521321 thru 0x521a03 (0x6e3 bytes) 000408[1] 1788823410.743335 write 0x5ad464 thru 0x5ad933 (0x4d0 bytes) 000411[1] 1788823410.835028 trunc from 0x9235e3 to 0x705d59 (0x21d889 bytes) 000413[0] 1788823410.878214 trunc from 0x705d5a to 0x37e70e (0x38764b bytes) 000414[1] 1788823410.961206 mapwrite 0x414efb thru 0x415399 (0x49f bytes) 000415[1] 1788823411.047977 punch from 0x277a65 to 0x27807a (0x616 bytes) 000416[1] 1788823411.078209 mapwrite 0x9db523 thru 0x9dc668 (0x1146 bytes) 000417[1] 1788823411.167029 write 0x7f433e thru 0x7f5284 (0xf47 bytes) 000418[0] 1788823411.202553 write 0x01a9f3 thru 0x01c47a (0x1a88 bytes) 000420[0] 1788823411.253272 punch from 0x951870 to 0x951a81 (0x212 bytes) 000421[1] 1788823411.273284 write 0x70f335 thru 0x710cab (0x1977 bytes) 000423[1] 1788823411.311515 mapwrite 0x5f4dd2 thru 0x5f567a (0x8a9 bytes) 000424[1] 1788823411.397390 punch from 0x52193f to 0x522584 (0xc46 bytes) 000425[0] 1788823411.414307 trunc from 0x9dc669 to 0x65e76f (0x37def9 bytes) 000426[1] 1788823411.470216 trunc from 0x65e770 to 0x04e575 (0x6101fa bytes) 000427[1] 1788823411.544751 trunc from 0x04e576 to 0x243a64 (0x1f54ef bytes) 000430[1] 1788823411.645219 write 0x45e1f9 thru 0x45ee54 (0xc5c bytes) 000431[0] 1788823411.671400 mapwrite 0x1a1b4a thru 0x1a20c5 (0x57c bytes) 000438[0] 1788823412.058508 mapwrite 0x5f7811 thru 0x5f9146 (0x1936 bytes) 000439[1] 1788823412.392494 mapwrite 0x97b18b thru 0x97c91c (0x1792 bytes) 000440[0] 1788823412.520555 write 0x8c98af thru 0x8ca748 (0xe9a bytes) 000441[0] 1788823412.575790 mapwrite 0x2e23eb thru 0x2e2df7 (0xa0d bytes) 000443[1] 1788823412.811315 write 0x85aafe thru 0x85b312 (0x815 bytes) 000446[1] 1788823412.997827 punch from 0x49fdb9 to 0x4a02f1 (0x539 bytes) 000447[1] 1788823413.004410 punch from 0x63bc5f to 0x63c1ae (0x550 bytes) 000448[0] 1788823413.019321 read 0x66f2f5 thru 0x66ff26 (0xc32 bytes) 000449[1] 1788823413.054221 mapwrite 0x7c820e thru 0x7c8819 (0x60c bytes) 000454[0] 1788823413.403072 read 0x0c7103 thru 0x0c728d (0x18b bytes) 000455[1] 1788823413.442190 punch from 0x5cecc1 to 0x5d025d (0x159d bytes) 000456[1] 1788823413.448453 punch from 0x4bdb19 to 0x4bf078 (0x1560 bytes) 000457[0] 1788823413.472246 read 0x7f1ac6 thru 0x7f2782 (0xcbd bytes) 000459[1] 1788823413.498747 mapwrite 0x72558a thru 0x726f5f (0x19d6 bytes) 000460[0] 1788823413.624193 write 0x6796ee thru 0x67b566 (0x1e79 bytes) 000461[0] 1788823413.690984 write 0x1e09d0 thru 0x1e29ad (0x1fde bytes) 000463[1] 1788823413.791768 mapwrite 0x15258a thru 0x152650 (0xc7 bytes) 000464[0] 1788823413.913399 mapread 0x6a885d thru 0x6aa6e6 (0x1e8a bytes) 000466[1] 1788823413.992989 read 0x96f666 thru 0x971407 (0x1da2 bytes) 000469[1] 1788823414.065330 mapread 0x1cfbf3 thru 0x1d18bc (0x1cca bytes) 000470[0] 1788823414.118073 mapwrite 0x7ec32a thru 0x7ecd70 (0xa47 bytes) 000471[1] 1788823414.202819 write 0x6bbadb thru 0x6bc9f2 (0xf18 bytes) 000474[1] 1788823414.276699 read 0x2bf9be thru 0x2bfc3d (0x280 bytes) 000475[1] 1788823414.305547 trunc from 0x97c91d to 0x9ce96d (0x52051 bytes) 000477[0] 1788823414.352137 punch from 0x02cbb9 to 0x02d9a7 (0xdef bytes) 000478[0] 1788823414.382513 mapread 0x8ebebf thru 0x8edda7 (0x1ee9 bytes) 000482[1] 1788823414.501224 trunc from 0x9ce96e to 0x648d4d (0x385c20 bytes) 000483[0] 1788823414.573267 mapwrite 0x633b37 thru 0x634604 (0xace bytes) 000484[1] 1788823414.732136 read 0x54d9da thru 0x54ef0f (0x1536 bytes) 000485[0] 1788823414.763499 punch from 0x2a8668 to 0x2aa367 (0x1d00 bytes) 000488[0] 1788823414.796305 write 0x651027 thru 0x65295e (0x1938 bytes) 000489[1] 1788823414.862525 trunc from 0x65295f to 0x2c2501 (0x39045d bytes) 000490[0] 1788823414.931176 read 0x194b54 thru 0x196657 (0x1b04 bytes) 000491[1] 1788823414.973413 trunc from 0x2c2502 to 0x3c8b72 (0x106671 bytes) 000493[1] 1788823415.055555 mapwrite 0x4eb231 thru 0x4ec8f2 (0x16c2 bytes) 000495[0] 1788823415.132105 mapread 0x321ee1 thru 0x322d4d (0xe6d bytes) 000497[0] 1788823415.188361 trunc from 0x4ec8f3 to 0x918570 (0x42bc7e bytes) 000499[1] 1788823415.268288 punch from 0x028661 to 0x029821 (0x11c1 bytes) All operations completed A-OK! PASS 16b (38s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 19:23:41 (1788823421) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (6s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 19:23:47 (1788823427) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.603369 s, 13.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.786932 s, 10.7 MB/s ldlm.namespaces.MGC192.168.201.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff88d8d124d000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff88d8e0eea800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff88d8d124d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff88d8e0eea800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff88d8d124d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff88d8e0eea800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 15.6337 s, 6.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.18533 s, 20.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.36622 s, 44.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.50463 s, 16.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9636 s, 9.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.49939 s, 23.3 MB/s PASS 16d (62s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 19:24:49 (1788823489) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.391031 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240649 s, 4.4 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 19:24:57 (1788823497) TIMEOUT PASS 16f (29s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 19:25:26 (1788823526) TIMEOUT PASS 16g (29s) == sanityn test 16h: mmap read after truncate file ======= 19:25:55 (1788823555) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (9s) == sanityn test 16i: read after truncate file ============ 19:26:05 (1788823565) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 19:26:13 (1788823573) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.91071 s, 9.0 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.44361 s, 18.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.56038 s, 10.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.13455 s, 12.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.02479 s, 8.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.26634 s, 11.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.97156 s, 8.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.28149 s, 20.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.83807 s, 14.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.60472 s, 7.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.75917 s, 7.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.7612 s, 9.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.87617 s, 9.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.29282 s, 8.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.41289 s, 7.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.09791 s, 8.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.27264 s, 8.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.60547 s, 7.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.70767 s, 7.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (41s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 19:26:54 (1788823614) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported Operation not supported truncating to largest ever: 0x232e1d3 skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1a7062c skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x32cce75 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! + fsx 1 done: rc=0 Mon Sep 7 19:27:47 EDT 2026 All operations completed A-OK! + fsx 2 done: rc=0 All operations completed A-OK! Mon Sep 7 19:27:48 EDT 2026 + fsx 3 done: rc=0 Mon Sep 7 19:27:48 EDT 2026 All operations completed A-OK! All operations completed A-OK! + fsx 4 done: rc=0 Mon Sep 7 19:27:49 EDT 2026 + fsx 5 done: rc=0 Mon Sep 7 19:27:49 EDT 2026 PASS 16k (63s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 19:27:57 (1788823677) fail_loc=0x8000030a PASS 17 (11s) == sanityn test 18: mmap sanity check =========================================================================================== 19:28:08 (1788823688) mmap test1: basic mmap operation (PASS, 0.12164s) mmap test2: MAP_PRIVATE not write back (PASS, 0.069206s) mmap test3: concurrent mmap ops on two nodes (PASS, 8.0643s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.6855s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.2723s) mmap test6: check mmap write/read content on two nodes (PASS, 0.24864s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.2919s) mmap test11: multi-thread shared mmap (PASS, 0.84635s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1129s) PASS 18 (25s) == sanityn test 19: test concurrent uncached read races ========================================================================= 19:28:33 (1788823713) oleg154-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg154-client: oleg154-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 19:28:38 (1788823718) PASS 20 (8s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 19:28:46 (1788823726) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (10s) == sanityn test 23: others should see updated atime while another read============================================================== 19:28:57 (1788823737) now is 1788823740 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 new atime is 1788823801 PASS 23 (72s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 19:30:08 (1788823808) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (7s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 19:30:15 (1788823815) PASS 24b (6s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 19:30:21 (1788823821) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (7s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 19:30:29 (1788823829) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 19:30:31 (1788823831) utime: good mknod times 1788823832 <= 1788823833 <= 1788823833 for /mnt/lustre/f26a utime: good mknod times 1788823832 <= 1788823833 <= 1788823833 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (7s) == sanityn test 26b: sync mtime between ost and mds ====== 19:30:39 (1788823839) PASS 26b (9s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 19:30:48 (1788823848) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-07 19:30:49.000000000 -0400 Birth: 2026-09-07 19:30:49.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-07 19:30:49.000000000 -0400 Birth: 2026-09-07 19:30:49.000000000 -0400 PASS 26c (7s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 19:30:56 (1788823856) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 19:31:05 (1788823865) fail_loc=0x314 PASS 31a (9s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 19:31:14 (1788823874) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg154-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.154@tcp:/lustre 7542784 39936 7496704 1% /mnt/lustre2 PASS 31b (25s) == sanityn test 31r: open-rename(replace) race =========== 19:31:39 (1788823899) -rw-r--r-- 1 root root 0 Sep 7 19:31 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 19:31:50 (1788823910) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43173 Terminated tail -f $tfile PASS 31s (8s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 19:31:58 (1788823918) PASS 31t (7s) == sanityn test 32b: lockless i/o ======================== 19:32:05 (1788823925) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (4s) == sanityn test 32c: contention detection ================ 19:32:10 (1788823930) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.46594 s, 35.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.32516 s, 39.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.21403 s, 16.3 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.46105 s, 15.1 MB/s contention_seconds 4, contention_events 21 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.64188 s, 19.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.89075 s, 18.1 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.77815 s, 18.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.87354 s, 18.2 MB/s PASS 32c (47s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 19:33:00 (1788823980) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 19:33:04 (1788823984) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 19:33:08 (1788823988) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 19:33:13 (1788823993) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (4s) == sanityn test 34: no lock timeout under IO ============= 19:33:16 (1788823996) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg154-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88d8d124d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88d8d124d000.ost_server_uuid in FULL state after 0 sec oleg154-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88d8d124d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88d8d124d000.ost_server_uuid in IDLE state after 0 sec oleg154-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88d8d124d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88d8d124d000.ost_server_uuid in FULL state after 0 sec oleg154-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88d8d124d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88d8d124d000.ost_server_uuid in IDLE state after 0 sec oleg154-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88d8d124d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88d8d124d000.ost_server_uuid in FULL state after 0 sec oleg154-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88d8d124d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88d8d124d000.ost_server_uuid in IDLE state after 0 sec PASS 34 (146s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 19:35:43 (1788824143) fail_loc=0x80000317 Race attempt 0 Wait for 50481 50535 for 60 sec... - open/close 647 (time 1788824161.46 total 10.00 last 64.70) - open/close 1263 (time 1788824171.46 total 20.00 last 61.59) - open/close 2189 (time 1788824181.47 total 30.01 last 92.52) - open/close 2882 (time 1788824191.48 total 40.02 last 69.26) - open/close 3591 (time 1788824201.49 total 50.03 last 70.83) total: 4000 open/close in 55.35 seconds: 72.27 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.201.154@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre 192.168.201.154@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre2 PASS 35 (74s) == sanityn test 36: handle ESTALE/open-unlink correctly == 19:36:57 (1788824217) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg154-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.63866 s, 32.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7498752) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.37746 s, 38.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7496704) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.58848 s, 33.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7492608) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.57062 s, 33.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.47726 s, 35.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7494656) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.29225 s, 40.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.34847 s, 38.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7494656) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.50652 s, 34.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7491584) after_dd(7443456) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.3039 s, 40.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7498752) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.38729 s, 37.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.08179 s, 25.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7494656) after_dd(7447552) after(7498752) PASS 36 (339s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 19:42:36 (1788824556) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 - create 1195 (time 1788824570.60 total 10.02 last 119.28) - create 2498 (time 1788824580.62 total 20.03 last 130.10) - create 3649 (time 1788824590.62 total 30.04 last 115.05) - create 4928 (time 1788824600.62 total 40.04 last 127.87) - create 6162 (time 1788824610.62 total 50.04 last 123.40) - create 7395 (time 1788824620.63 total 60.05 last 123.22) - create 8588 (time 1788824630.64 total 70.05 last 119.23) - create 9952 (time 1788824640.64 total 80.06 last 136.33) total: 10000 create in 80.37 seconds: 124.43 ops/second PASS 37 (93s) == sanityn test 39a: file mtime does not change after rename ========================================================== 19:44:09 (1788824649) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 19:44:17 (1788824657) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 19:44:26 (1788824666) repeat after cancel_lru_locks PASS 39c (8s) == sanityn test 39d: sync write should update mtime ====== 19:44:34 (1788824674) fail_loc=0x411 fail_loc=0 PASS 39d (8s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 19:44:44 (1788824684) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (18s) == sanityn test 40c: pdirops: link and others ======================================================================== 19:45:02 (1788824702) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (15s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 19:45:18 (1788824718) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (16s) == sanityn test 40e: pdirops: rename and others ======================================================================== 19:45:34 (1788824734) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (18s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 19:45:52 (1788824752) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (14s) == sanityn test 41b: pdirops: create vs create ======================================================================== 19:46:07 (1788824767) open: File exists Conflict PASS 41b (16s) == sanityn test 41c: pdirops: create vs link ======================================================================== 19:46:23 (1788824783) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (15s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 19:46:38 (1788824798) Conflict PASS 41d (15s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 19:46:54 (1788824814) Conflict PASS 41e (13s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 19:47:07 (1788824827) Conflict PASS 41f (12s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 19:47:20 (1788824840) Conflict PASS 41g (12s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 19:47:32 (1788824852) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (14s) == sanityn test 41i: reint_open: create vs create ======== 19:47:46 (1788824866) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1040s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 20:05:06 (1788825906) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (10s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 20:05:16 (1788825916) open: File exists Conflict PASS 42b (10s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 20:05:27 (1788825927) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (10s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 20:05:37 (1788825937) Conflict PASS 42d (11s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 20:05:48 (1788825948) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (10s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 20:05:58 (1788825958) Conflict PASS 42f (11s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 20:06:10 (1788825970) Conflict PASS 42g (11s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 20:06:21 (1788825981) Conflict PASS 42h (10s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 20:06:31 (1788825991) PASS 43a (57s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 20:07:28 (1788826048) Conflict PASS 43b (10s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 20:07:38 (1788826058) Conflict PASS 43c (11s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 20:07:49 (1788826069) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (10s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 20:07:59 (1788826079) Conflict PASS 43e (9s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 20:08:08 (1788826088) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (10s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 20:08:18 (1788826098) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (9s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 20:08:28 (1788826108) Conflict PASS 43h (9s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 20:08:37 (1788826117) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (2s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 20:08:39 (1788826119) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (77s) == sanityn test 43k: unlink vs create ==================== 20:09:57 (1788826197) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1016s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 20:26:52 (1788827212) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Mon Sep 7 20:26:59 EDT 2026 PASS 44a (10s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 20:27:02 (1788827222) open: File exists Conflict PASS 44b (10s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 20:27:12 (1788827232) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (8s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 20:27:21 (1788827241) Conflict PASS 44d (9s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 20:27:29 (1788827249) Conflict PASS 44e (10s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 20:27:39 (1788827259) Conflict PASS 44f (10s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 20:27:49 (1788827269) Conflict PASS 44g (10s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 20:27:59 (1788827279) Conflict PASS 44h (9s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 20:28:09 (1788827289) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (2s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 20:28:11 (1788827291) PASS 45a (88s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 20:29:39 (1788827379) Conflict PASS 45b (10s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 20:29:49 (1788827389) Conflict PASS 45c (9s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 20:29:58 (1788827398) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (9s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 20:30:07 (1788827407) Conflict PASS 45e (10s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 20:30:17 (1788827417) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (9s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 20:30:27 (1788827427) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (9s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 20:30:36 (1788827436) Conflict PASS 45h (10s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 20:30:46 (1788827446) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (2s) == sanityn test 45j: read vs rename ====================== 20:30:48 (1788827448) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (971s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 20:46:59 (1788828419) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (10s) == sanityn test 46b: pdirops: link vs create ======================================================================== 20:47:09 (1788828429) open: File exists Conflict PASS 46b (10s) == sanityn test 46c: pdirops: link vs link ======================================================================== 20:47:19 (1788828439) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (10s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 20:47:29 (1788828449) Conflict PASS 46d (11s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 20:47:40 (1788828460) Conflict PASS 46e (10s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 20:47:50 (1788828470) Conflict PASS 46f (9s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 20:48:00 (1788828480) Conflict PASS 46g (10s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 20:48:10 (1788828490) Conflict PASS 46h (12s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 20:48:22 (1788828502) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (2s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 20:48:24 (1788828504) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (2s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 20:48:27 (1788828507) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (2s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 20:48:29 (1788828509) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 20:48:32 (1788828512) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (2s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 20:48:34 (1788828514) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 20:48:37 (1788828517) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 20:48:40 (1788828520) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 20:48:43 (1788828523) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0573564 s, 179 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (10s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 20:48:53 (1788828533) 0+1 records in 0+1 records out 158 bytes copied, 0.0313606 s, 5.0 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218788) - No such process PASS 51a (8s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 20:49:01 (1788828541) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0156758 s, 65.3 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045486 s, 23.1 MB/s 3145728 PASS 51b (19s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 20:49:20 (1788828560) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0509026 s, 20.1 kB/s PASS 51c (11s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 20:49:31 (1788828571) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389383 s, 26.9 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (7s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 20:49:38 (1788828578) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 read lease(1) has applied. PASS 51e (7s) == sanityn test 54: rename locking ======================= 20:49:45 (1788828585) llite.lustre-ffff88d8d124d000.xattr_cache=1 llite.lustre-ffff88d8e0eea800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:49:46.000000000 -0400 Modify: 2026-09-07 20:49:46.000000000 -0400 Change: 2026-09-07 20:49:46.000000000 -0400 Birth: 2026-09-07 20:49:46.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:49:52.000000000 -0400 Modify: 2026-09-07 20:49:53.000000000 -0400 Change: 2026-09-07 20:49:53.000000000 -0400 Birth: 2026-09-07 20:49:53.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:49:58.000000000 -0400 Modify: 2026-09-07 20:49:58.000000000 -0400 Change: 2026-09-07 20:49:58.000000000 -0400 Birth: 2026-09-07 20:49:59.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:49:58.000000000 -0400 Modify: 2026-09-07 20:49:59.000000000 -0400 Change: 2026-09-07 20:49:59.000000000 -0400 Birth: 2026-09-07 20:49:59.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:50:04.000000000 -0400 Modify: 2026-09-07 20:50:05.000000000 -0400 Change: 2026-09-07 20:50:05.000000000 -0400 Birth: 2026-09-07 20:50:05.000000000 -0400 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505022 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 20:50:04.000000000 -0400 Modify: 2026-09-07 20:50:05.000000000 -0400 Change: 2026-09-07 20:50:05.000000000 -0400 Birth: 2026-09-07 20:50:05.000000000 -0400 PASS 54 (29s) == sanityn test 55a: rename vs unlink target dir ========= 20:50:14 (1788828614) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (11s) == sanityn test 55b: rename vs unlink source dir ========= 20:50:25 (1788828625) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (11s) == sanityn test 55c: rename vs unlink orphan target dir == 20:50:36 (1788828636) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (17s) == sanityn test 55d: rename file vs link ================= 20:50:53 (1788828653) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (13s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 20:51:06 (1788828666) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (2s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 20:51:08 (1788828668) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (17s) == sanityn test 55g: rename: race with trylock =========== 20:51:25 (1788828685) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (18s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 20:51:44 (1788828704) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788828705 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.803146 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (81s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 20:53:05 (1788828785) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788828788 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.60624 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (85s) == sanityn test 60: Verify data_version behaviour ======== 20:54:30 (1788828870) PASS 60 (5s) == sanityn test 70a: cd directory && rm directory ======== 20:54:35 (1788828875) PASS 70a (7s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 20:54:42 (1788828882) PASS 70b (7s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 20:54:49 (1788828889) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0137164 s, 4.8 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 20:54:52 (1788828892) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00970205 s, 6.8 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (2s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 20:54:55 (1788828895) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (2s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 20:54:57 (1788828897) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0112065 s, 5.8 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (2s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 20:55:00 (1788828900) llite.lustre-ffff88d8d124d000.xattr_cache=1 llite.lustre-ffff88d8e0eea800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (5s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 20:55:04 (1788828904) llite.lustre-ffff88d8d124d000.xattr_cache=1 llite.lustre-ffff88d8e0eea800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (6s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 20:55:10 (1788828910) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0212461 s, 48.2 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0212934 s, 48.1 kB/s 231582: get lock2 231583: get lock1 231583: done 231583: get lock2 231582: done 231582: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 231582: cannot get lock1: Resource deadlock avoided 231583: done 231583: exit rc=0 231582: done 231582: exit rc=0 PASS 74 (9s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 20:55:19 (1788828919) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0553105 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244151 s, 42.9 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232164 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (9s) == sanityn test 76: Verify MDT open_files listing ======== 20:55:28 (1788828928) opening files: ........ closing files: ........ PASS 76 (48s) == sanityn test 77a: check FIFO NRS policy =============== 20:56:16 (1788828976) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.334366 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577011 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773754 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645833 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733243 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941471 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182571 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184806 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164066 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177247 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165069 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220699 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230116 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236686 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20653 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24845 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8e0eea800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff88d8e0eea800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236866 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027463 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299312 s, 35.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266807 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550562 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375271 s, 27.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540758 s, 19.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360694 s, 29.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368625 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480157 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400465 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041527 s, 25.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513441 s, 20.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529764 s, 19.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8e0eea800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631711 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492624 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053732 s, 19.5 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (7s) == sanityn test 77b: check CRR-N NRS policy ============== 20:56:23 (1788828983) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.312032 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517698 s, 20.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420576 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443719 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901748 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403073 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744085 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149037 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770127 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169764 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107344 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149939 s, 7.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131457 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180447 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181846 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178996 s, 5.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8e0eea800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88d8e0eea800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222112 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198789 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00539527 s, 194 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240566 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.006555 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201566 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021315 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101247 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699573 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353267 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207491 s, 50.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405655 s, 25.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766736 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616456 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629939 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00366633 s, 286 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463082 s, 22.6 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.306893 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523977 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075156 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181837 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171678 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213889 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190233 s, 5.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168008 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212398 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197862 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220893 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177426 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202695 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216217 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206825 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218869 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8e0eea800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff88d8e0eea800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197633 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227533 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188645 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00373694 s, 281 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00279746 s, 375 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00417083 s, 251 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591608 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00284312 s, 369 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105845 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841197 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951598 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496835 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409838 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391091 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402827 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308992 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205556 s, 51.0 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (9s) == sanityn test 77c: check ORR NRS policy ================ 20:56:32 (1788828992) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.303366 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480653 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639814 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128575 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192295 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192487 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208409 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211597 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198127 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207256 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196834 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233668 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261171 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256958 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279022 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264151 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290071 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8e0eea800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff88d8e0eea800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00776386 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426189 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324217 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371408 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312443 s, 33.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406246 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495158 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266111 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438844 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528518 s, 19.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474219 s, 22.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503577 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509759 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521542 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484245 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588369 s, 17.8 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.295051 s, 56.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495283 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806769 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721455 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132291 s, 7.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8e0eea800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff88d8e0eea800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235818 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228741 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222317 s, 4.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212543 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14991 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167343 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229109 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21694 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183949 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234064 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261433 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224238 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8e0eea800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff88d8e0eea800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646688 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685109 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808466 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107208 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127175 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110017 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838744 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103673 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152514 s, 68.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100206 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123627 s, 84.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696529 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485753 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447071 s, 23.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291372 s, 36.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366741 s, 28.6 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.334121 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480701 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065403 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670822 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099159 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116386 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124112 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143934 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1276 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167318 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144474 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118583 s, 8.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8e0eea800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff88d8e0eea800.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139744 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155389 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191818 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176632 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207324 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936684 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518272 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770439 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00258131 s, 406 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722287 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097914 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490055 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342637 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468695 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309205 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276503 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253643 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229883 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130823 s, 80.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016467 s, 63.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211943 s, 49.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (11s) == sanityn test 77d: check TRR nrs policy ================ 20:56:43 (1788829003) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.411397 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049732 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691714 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140096 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230397 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243508 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250766 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204318 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259357 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249147 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255735 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235448 s, 4.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431396 s, 24.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655375 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354587 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296191 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519903 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293094 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335833 s, 31.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277748 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840319 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351784 s, 29.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125892 s, 83.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144018 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966659 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304872 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157022 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473503 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161945 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375822 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8e0eea800.lock_unused_count=10 ldlm.namespaces.lustre-OST0001-osc-ffff88d8e0eea800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119762 s, 87.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00451029 s, 232 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.333793 s, 50.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702165 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848322 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232948 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169922 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209872 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203005 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229024 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216893 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233768 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210102 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191118 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203178 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200098 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234374 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19173 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248879 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8e0eea800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff88d8e0eea800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424464 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070098 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378927 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070838 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423092 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777967 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811338 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471796 s, 22.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547763 s, 19.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597974 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526298 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663064 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697616 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615303 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690013 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730318 s, 14.4 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.477925 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431229 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743386 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561021 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740256 s, 14.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140899 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104701 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108585 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149398 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196955 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193196 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173564 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569695 s, 18.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8e0eea800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff88d8e0eea800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211033 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685951 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641333 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655403 s, 16.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8e0eea800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00455986 s, 230 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368219 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00788394 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058283 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019822 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00449863 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00767514 s, 137 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065887 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076517 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116799 s, 89.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109418 s, 95.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172236 s, 60.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687848 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529925 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547035 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932073 s, 11.2 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (12s) == sanityn test 77e: check TBF NID nrs policy ============ 20:56:55 (1788829015) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.201.54@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.291025 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851414 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12319 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134967 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159576 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131658 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171862 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176999 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16089 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185985 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182768 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136912 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17006 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169332 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164794 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123569 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211112 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178265 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189379 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177101 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187838 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189568 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194446 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203912 s, 5.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211599 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196373 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209837 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198678 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20058 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201908 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202951 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211087 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244347 s, 4.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.468359 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476493 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082292 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758723 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053277 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720339 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358485 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122598 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135576 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131935 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191237 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151592 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173994 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132469 s, 7.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8e0eea800.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111994 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169072 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220391 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960771 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13447 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132222 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12506 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110632 s, 9.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113051 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113146 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115877 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120822 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137042 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126535 s, 8.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126787 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164238 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135994 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148741 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143326 s, 7.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.295525 s, 56.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741609 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766735 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145456 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101118 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183762 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21225 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215198 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207275 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157094 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18924 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200391 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238805 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20503 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242502 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295733 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8e0eea800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff88d8e0eea800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33791 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687753 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360338 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354971 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608334 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578162 s, 18.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495919 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579604 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630878 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736934 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565473 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598715 s, 17.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622415 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714124 s, 14.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666266 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840865 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080519 s, 13.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.483027 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955018 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145361 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168266 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200266 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191266 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318051 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319356 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258607 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281388 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371272 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310369 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357067 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287827 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285075 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218543 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426911 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8e0eea800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88d8e0eea800.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021404 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554653 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061279 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356149 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041136 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431209 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404446 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482433 s, 21.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703653 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718543 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844148 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076382 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563034 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707505 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732178 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912141 s, 11.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (21s) == sanityn test 77f: check TBF JobID nrs policy ========== 20:57:17 (1788829037) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.602651 s, 27.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109726 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16447 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136088 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124025 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210392 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687215 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229597 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243462 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176675 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17968 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332309 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.428997 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366409 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.512906 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392516 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.737308 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187781 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102039 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200982 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113611 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11835 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175143 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123107 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106389 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141475 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283636 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224143 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0042056 s, 249 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229145 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139704 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282265 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155379 s, 67.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.513759 s, 32.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501529 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07696 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648611 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138983 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174959 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975497 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109169 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293455 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32648 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370501 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365981 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321007 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409202 s, 2.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342693 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322314 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324699 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00766193 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00946976 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16139 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213652 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674734 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109745 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144923 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336786 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183116 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124102 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057441 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893281 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094219 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00306322 s, 342 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517843 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701159 s, 15.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.448021 s, 37.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743683 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720429 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490548 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177563 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860336 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205606 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210076 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158972 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265206 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169431 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26778 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8e0eea800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff88d8e0eea800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206652 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342473 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312047 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286807 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346208 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8e0eea800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918099 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954574 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993984 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112558 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721148 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106626 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095593 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922088 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936344 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954564 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928832 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896566 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090686 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110878 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111744 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122635 s, 8.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.274976 s, 61.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072401 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967571 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126542 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13132 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142122 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188749 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205804 s, 5.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227781 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210543 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207112 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22019 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234778 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223866 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221919 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267865 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283062 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329546 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00403711 s, 260 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422879 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150866 s, 69.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564115 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00257499 s, 407 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00269815 s, 389 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875446 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542798 s, 19.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423947 s, 24.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471279 s, 22.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00960132 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284247 s, 36.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225857 s, 46.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158378 s, 66.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215925 s, 48.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (24s) == sanityn test 77g: Change TBF type directly ============ 20:57:41 (1788829061) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (9s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 20:57:50 (1788829070) oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg154-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg154-client: oleg154-server: ssh exited with exit code 2 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg154-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg154-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg154-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 PASS 77h (10s) == sanityn test 77i: Change rank of TBF rule ============= 20:58:00 (1788829080) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (18s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 20:58:18 (1788829098) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.44874 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503051 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521714 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127278 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134225 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155564 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193502 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162191 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186598 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177573 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18119 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232609 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210297 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224106 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195646 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236185 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211109 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05071 s, 20.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852893 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285901 s, 36.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322469 s, 32.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342437 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824158 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936782 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657022 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810859 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620961 s, 16.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629318 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638316 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736491 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755438 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767871 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786348 s, 13.3 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.3492 s, 16.5 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5698 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (61s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 20:59:19 (1788829159) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg154-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg154-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg154-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg154-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28398 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07013 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101814 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102878 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130287 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658722 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166513 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116314 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133007 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111026 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160216 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163478 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204341 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258557 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269285 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275865 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514558 s, 20.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381514 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545426 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563347 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571555 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578473 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578135 s, 18.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412093 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406804 s, 25.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404495 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528013 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472551 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049064 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569091 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574646 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772973 s, 13.6 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9053 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9111 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg154-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg154-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg154-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg154-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29434 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638286 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735535 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135207 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149142 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131999 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14722 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825848 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149553 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145284 s, 7.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120273 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23099 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195716 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216581 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246371 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232428 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189368 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370919 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359908 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400781 s, 26.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424087 s, 24.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444314 s, 23.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416016 s, 25.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517931 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566096 s, 18.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556756 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513807 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544258 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482079 s, 21.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558915 s, 18.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499796 s, 21.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522046 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514229 s, 20.4 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8872 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9428 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (146s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 21:01:45 (1788829305) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg154-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg154-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg154-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg154-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27132 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333667 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668654 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813544 s, 12.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829372 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113118 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124748 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121701 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10675 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154599 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200992 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230383 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239791 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198905 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262031 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275379 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235605 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434482 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419198 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385925 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492207 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528848 s, 19.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462257 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358415 s, 29.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540481 s, 19.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042405 s, 24.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619292 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362999 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392988 s, 26.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052128 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476301 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620733 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694242 s, 15.1 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9285 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9297 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg154-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg154-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg154-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg154-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27216 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883508 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107691 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987657 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105621 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120043 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116126 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111662 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167941 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162035 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21448 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22038 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220238 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169154 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22529 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23606 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242163 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808411 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869118 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804468 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894267 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758309 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974516 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103954 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107565 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107209 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104532 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937571 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1027 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1049 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999053 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104711 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109141 s, 9.6 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8404 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9108 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (144s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 21:04:09 (1788829449) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.484263 s, 34.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623121 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070705 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194841 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203734 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146032 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219441 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203591 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165754 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177613 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196124 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178957 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216008 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262541 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240309 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206612 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211845 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062042 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380146 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719079 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375453 s, 27.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480461 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500765 s, 20.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514836 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565053 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950227 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589228 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694618 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703386 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073552 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771399 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765191 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845713 s, 12.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.25333 s, 16.8 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87755 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.54@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.54@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.415515 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418124 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893285 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125096 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189576 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184801 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189112 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210707 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216048 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206537 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193918 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225456 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241504 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253594 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245925 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238835 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211477 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332883 s, 31.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518762 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254734 s, 41.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343104 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372649 s, 28.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380871 s, 27.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038918 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312026 s, 33.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350717 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431207 s, 24.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443769 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458107 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463251 s, 22.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385228 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530663 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475325 s, 22.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.77344 s, 18.2 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84502 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.54@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.424129 s, 39.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408925 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082384 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472996 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867268 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084934 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181745 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191092 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117079 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203421 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186919 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226958 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174893 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217903 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164663 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155834 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273834 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435749 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365618 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481617 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421082 s, 24.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447591 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471503 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675615 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457103 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466136 s, 22.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544033 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749472 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519545 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581589 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588191 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589495 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656204 s, 16.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.00242 s, 17.5 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.08911 s, 20.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.473286 s, 35.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311093 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690581 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699816 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067871 s, 15.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928443 s, 11.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696466 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055782 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500354 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697329 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729383 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162801 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155056 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355953 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322396 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.4307 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.438045 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400291 s, 26.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380149 s, 27.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376584 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582656 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191174 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491937 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260244 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628682 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511612 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035139 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378848 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599317 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616393 s, 17.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042985 s, 24.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435847 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471694 s, 22.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96534 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84035 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.95511 s, 17.6 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83874 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.94077 s, 17.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477994 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062551 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209515 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824343 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103378 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975936 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151548 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125996 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157119 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144156 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2281 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149532 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236014 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142102 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234083 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215417 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421703 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274752 s, 38.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281629 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530862 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302683 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350185 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395139 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413637 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385135 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436788 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041843 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438611 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426547 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532897 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796422 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810123 s, 12.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7033 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.56 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.418035 s, 40.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061718 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662183 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107339 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165387 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168112 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180258 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178293 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175134 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188101 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199158 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206757 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175544 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194566 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187198 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185702 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219172 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316808 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719608 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512613 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511629 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358332 s, 29.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379759 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455964 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570053 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496867 s, 21.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067821 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552986 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756488 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778386 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539285 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575148 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686181 s, 15.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.36707 s, 16.5 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86896 s, 10.6 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.606552 s, 27.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347719 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922298 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137136 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164456 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169456 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173054 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108709 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668997 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991772 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172454 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221507 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14145 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149832 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209039 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18773 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247358 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469025 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216857 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468725 s, 22.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044701 s, 23.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224012 s, 46.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488608 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293313 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275629 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283212 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041108 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340432 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346602 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679161 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413956 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525799 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456352 s, 23.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0439 s, 10.4 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84036 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96506 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83056 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo PASS 77k (406s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 21:10:55 (1788829855) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.310451 s, 54.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584966 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378052 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606572 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786706 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707783 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738736 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671406 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102592 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774894 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147361 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172249 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175124 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194433 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185452 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173828 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194073 s, 5.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279961 s, 37.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335071 s, 31.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308893 s, 33.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385319 s, 27.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369478 s, 28.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300388 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336941 s, 31.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03385 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300342 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325151 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384059 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322919 s, 32.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340828 s, 30.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340997 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338493 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396168 s, 26.5 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.0213 s, 20.9 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78874 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kb (45s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 21:11:40 (1788829900) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.54@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.54@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.349076 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391684 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395779 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121052 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139442 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140016 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145068 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151352 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13593 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144704 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149561 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150544 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151793 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169386 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150723 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211812 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202139 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213485 s, 49.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019713 s, 53.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218671 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335221 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245238 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297743 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301323 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034234 s, 30.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377643 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344474 s, 30.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039964 s, 26.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375561 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042451 s, 24.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043691 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433411 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431077 s, 24.3 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.34466 s, 19.6 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84729 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kc (47s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 21:12:27 (1788829947) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.54@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.337453 s, 49.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315498 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327572 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174192 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171333 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180385 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174929 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186161 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157853 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174095 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167308 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188947 s, 5.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177459 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1609 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186381 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176405 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221065 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02547 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391371 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033713 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283909 s, 36.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359983 s, 29.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377324 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302138 s, 34.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364828 s, 28.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304086 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438697 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418771 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431113 s, 24.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401454 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318164 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438575 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043455 s, 24.1 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.0767 s, 20.7 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96003 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_policies=fifo PASS 77kd (41s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 21:13:08 (1788829988) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.617367 s, 27.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318146 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465135 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975684 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122708 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997954 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109966 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136998 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122958 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140809 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136065 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157116 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164234 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171309 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165721 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19516 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194026 s, 5.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322815 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314137 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655025 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724323 s, 14.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509325 s, 20.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047685 s, 22.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393825 s, 26.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490468 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742267 s, 14.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482487 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555133 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704233 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898964 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086161 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590572 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058776 s, 17.8 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0151 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90544 s, 10.6 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.28187 s, 19.9 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79656 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77ke (89s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 21:14:37 (1788830077) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.90321 s, 18.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578713 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082778 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581119 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942514 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491548 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103267 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443367 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795755 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047162 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100667 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156744 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178962 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151193 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188604 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176343 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20852 s, 5.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440718 s, 23.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477994 s, 21.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418824 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342214 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342477 s, 30.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632091 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406094 s, 25.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428653 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436187 s, 24.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346158 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369453 s, 28.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488047 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386059 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399887 s, 26.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467825 s, 22.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475612 s, 22.0 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6936 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9089 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_policies=fifo PASS 77kf (72s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 21:15:49 (1788830149) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.35177 s, 47.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314242 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397656 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110184 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111794 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135405 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127228 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144239 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153343 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161139 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165573 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162305 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169906 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150777 s, 7.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169991 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165944 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166921 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349622 s, 30.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324545 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283655 s, 37.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340939 s, 30.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268113 s, 39.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416645 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200059 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232942 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369713 s, 28.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288923 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278714 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340747 s, 30.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299944 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289827 s, 36.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273787 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316053 s, 33.2 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.0557 s, 20.7 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8021 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.481654 s, 34.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492495 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649693 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091955 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123727 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143353 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164779 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148435 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167813 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132386 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122886 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152805 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125516 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178036 s, 5.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150789 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185265 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221831 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267501 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455424 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00423934 s, 247 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463536 s, 22.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590618 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293979 s, 35.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030942 s, 33.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310322 s, 33.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423065 s, 24.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667649 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372908 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446672 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00986462 s, 106 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395851 s, 26.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00870533 s, 120 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164589 s, 63.7 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91172 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8267 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92116 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0599 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77kg (143s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 21:18:12 (1788830292) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre opts: Stopping clients: oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg154-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg154-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg154-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg154-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.54,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg154-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.54,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg154-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.54,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69781/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.54,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg154-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: kmemleak disabled oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: MODOPTS_LIBCFS= oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre Starting client oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre Started clients oleg154-client.virtnet: 192.168.201.154@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre2 Starting client oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre2 Started clients oleg154-client.virtnet: 192.168.201.154@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88d8d1449000.idle_timeout=debug osc.lustre-OST0000-osc-ffff88d8d275c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88d8d1449000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88d8d275c000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.610387 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535249 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858738 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889985 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102915 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122832 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111509 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256223 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27889 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202097 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20199 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320099 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312856 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313337 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306378 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332863 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344222 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8d1449000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff88d8d1449000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24494 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257872 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23614 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247743 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26549 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238617 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276969 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225706 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277326 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267343 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266587 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276466 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272365 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277737 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279164 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288598 s, 3.6 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.70666 s, 13.6 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.54654 s, 18.9 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.603304 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100779 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127464 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140672 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138233 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186744 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189152 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205572 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197359 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219186 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316953 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347075 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265348 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378327 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375084 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390406 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30589 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8d1449000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff88d8d1449000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770459 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512394 s, 20.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122126 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647575 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982125 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523874 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723595 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726157 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734556 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143241 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924223 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898702 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905832 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903734 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157409 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111875 s, 9.4 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.18654 s, 12.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98299 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (218s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 21:21:50 (1788830510) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg154-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (19s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 21:22:10 (1788830530) On MGS 192.168.201.154, active = nodemap.active=1 On MGS 192.168.201.154, TBF.id = nodemap.TBF.id=1 ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.613987 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783909 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842322 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120256 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095324 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552804 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171381 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201609 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2113 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23707 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181915 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26531 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155835 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291701 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311943 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224286 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286869 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8d1449000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88d8d1449000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825621 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880892 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698162 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682715 s, 15.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983468 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647247 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00739655 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018598 s, 56.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8d1449000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112581 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927771 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034915 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432436 s, 24.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094493 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231361 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354978 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222252 s, 47.2 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.06828 s, 13.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.31756 s, 19.7 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.201.154, TBF.id = On MGS 192.168.201.154, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.599367 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063439 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910093 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925005 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113949 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949314 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170182 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220581 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221279 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24517 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233365 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260696 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310926 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249632 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303216 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278927 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355851 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705866 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209473 s, 50.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279689 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054894 s, 19.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8d1449000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527108 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709392 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105499 s, 99.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743722 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676542 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068758 s, 15.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00810348 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714794 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754902 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506233 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348478 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491596 s, 21.3 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.49086 s, 12.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.69734 s, 38.9 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg154-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg154-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 On MGS 192.168.201.154, active = nodemap.active=1 On MGS 192.168.201.154, TBF.id = nodemap.TBF.id=2 ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.518128 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624458 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072343 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486992 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714811 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158102 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128255 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123358 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122073 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13196 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265973 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195702 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145377 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199985 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194431 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272079 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24339 s, 4.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8d1449000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281442 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421256 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150286 s, 69.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00550492 s, 190 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361029 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00547404 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00833184 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426389 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705487 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00242064 s, 433 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607357 s, 17.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817748 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582445 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505272 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501123 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136595 s, 76.8 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.32713 s, 14.3 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97944 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start nmdyn nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.667127 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761667 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125001 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150615 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142325 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119151 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143282 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983481 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280608 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249809 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334958 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308644 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262797 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304505 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354688 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.435054 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.470008 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8d1449000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209969 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21725 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240766 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180618 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173899 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15123 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190111 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160116 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199161 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198595 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173712 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173846 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174529 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278989 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177853 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185885 s, 5.6 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.86937 s, 15.3 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.27263 s, 19.9 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nmdyn ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.201.154, TBF.id = On MGS 192.168.201.154, active = nodemap.active=0 Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.201.154, active = nodemap.active=0 PASS 77kj (239s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 21:26:09 (1788830769) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (9s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 21:26:18 (1788830778) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg154-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.94213 s, 266 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg154-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.1408 s, 261 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (59s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 21:27:17 (1788830837) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.07136 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169748 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190112 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360143 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274029 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.448982 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373363 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.494836 s, 2.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00939863 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982983 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283828 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.561277 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.558328 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109229 s, 96.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.598247 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.646847 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846765 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.565064 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00239724 s, 437 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00236995 s, 442 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.670851 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.719592 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00222977 s, 470 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286709 s, 36.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028889 s, 36.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00449506 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167476 s, 62.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00675277 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316277 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.675953 s, 1.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730499 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446673 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.758196 s, 1.4 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.5942 s, 8.3 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.6704 s, 18.5 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.702515 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106292 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808756 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139306 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122227 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15852 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686333 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224206 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157002 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209055 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226899 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194198 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207358 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327225 s, 3.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336875 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308992 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8d1449000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88d8d1449000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.474636 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195544 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203719 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216912 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213869 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264071 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218564 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222664 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247131 s, 4.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88d8d1449000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248089 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258831 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255994 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278251 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278664 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303598 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284195 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303495 s, 3.5 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.84611 s, 13.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.42356 s, 19.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (104s) == sanityn test 77o: Changing rank should not panic ====== 21:29:01 (1788830941) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (12s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 21:29:13 (1788830953) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (117s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 21:31:10 (1788831070) mds.MDS.mdt.nrs_policies=tbf oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg154-client: oleg154-server: ssh exited with exit code 36 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (42s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 21:31:53 (1788831113) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 276929 kill md_thread xargs PASS 77r (58s) == sanityn test 77s: Check TBF LRU shrinker ============== 21:32:51 (1788831171) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, seq 2, type=jobid, ref 10 touch jobid={touch.*} 1000, seq 2, type=jobid, ref 0 unlink jobid={unlink.*} 1000, seq 3, type=jobid, ref 10 unlink jobid={unlink.*} 1000, seq 3, type=jobid, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (72s) == sanityn test 77ta: Testing for dynamic|static classify parameter setting for TBF ========================================================== 21:34:03 (1788831243) ost.OSS.ost_io.nrs_policies=fifo oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid+uid classify=foo: Invalid argument oleg154-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid+uid classify=foo': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid+uid foo=static: Invalid argument oleg154-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid+uid foo=static': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid+uid classify=: Invalid argument oleg154-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid+uid classify=': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic ost.OSS.ost_io.nrs_policies=tbf nid classify=static ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf reg nid classify=dynamic ost.OSS.ost_io.nrs_policies=tbf hp nid ost.OSS.ost_io.nrs_tbf_rule=start mixed1 nid={0@lo} rate=100 ost.OSS.ost_io.nrs_tbf_rule=stop mixed1 ost.OSS.ost_io.nrs_policies=tbf reg nid classify=static ost.OSS.ost_io.nrs_policies=tbf hp nid classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start mixed2 nid={0@lo} rate=100 ost.OSS.ost_io.nrs_tbf_rule=stop mixed2 ost.OSS.ost_io.nrs_policies=tbf reg nid+uid ost.OSS.ost_io.nrs_policies=tbf hp nid oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start mixed3 nid={0@lo}&uid={500} rate=100: Invalid argument oleg154-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start mixed3 nid={0@lo}&uid={500} rate=100': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=tbf nid classify=static classify=dynamic ost.OSS.ost_io.nrs_policies=tbf reg nid classify=dynamic ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=dynamic state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=static classify=dynamic state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf hp nid classify=static ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf state: stopped fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=static state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_policies=fifo PASS 77ta (44s) == sanityn test 77tB: Request falling to default rule should not panic for dynamic TBF ========================================================== 21:34:47 (1788831287) ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8829 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9509 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77tB.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=start unmatchrule nid={192.0.2.1@tcp} rate=100 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9362 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.971 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77tB.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop unmatchrule ost.OSS.ost_io.nrs_policies=fifo PASS 77tB (144s) == sanityn test 77th: TBF: a mixed classify head pair keeps each head type ========================================================== 21:37:12 (1788831432) ost.OSS.ost_io.nrs_policies=tbf reg nid+opcode ost.OSS.ost_io.nrs_policies=tbf hp nid classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start mixedhp nid={0@lo} rate=100 regular_requests: CPT 0: mixedhp nid={0@lo} 100, seq 2, type=nid+opcode, ref 0 default * 10000, seq 1, type=nid+opcode, ref 0 high_priority_requests: CPT 0: mixedhp nid={0@lo} 100, seq 2, type=nid, ref 0 default * 10000, seq 1, type=nid, ref 0 ost.OSS.ost_io.nrs_tbf_rule=stop mixedhp ost.OSS.ost_io.nrs_policies=fifo PASS 77th (15s) == sanityn test 77tb: Testing for dynamic classify TBF Scheduler ========================================================== 21:37:27 (1788831447) ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic ost.OSS.ost_io.nrs_policies= regular_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=dynamic state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 high_priority_requests: - name: fifo state: started fallback: yes queued: 0 active: 0 - name: crrn state: stopped fallback: no queued: 0 active: 0 - name: orr state: stopped fallback: no queued: 0 active: 0 - name: trr state: stopped fallback: no queued: 0 active: 0 - name: tbf nid classify=dynamic state: started fallback: no queued: 0 active: 0 - name: delay state: stopped fallback: no queued: 0 active: 0 ost.OSS.ost_io.nrs_tbf_classify= reg_classify:dynamic hp_classify:dynamic ost.OSS.ost_io.nrs_tbf_rule=start nidall nid={0@lo 192.168.201.54@tcp} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo 192.168.201.54@tcp}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start nidwr nid={0@lo 192.168.201.54@tcp}&opcode={ost_write} rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: nidwr nid={0@lo 192.168.201.54@tcp}&opcode={ost_write} 15, seq 4, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.201.54@tcp}&opcode={ost_read} 5, seq 3, type=nid+opcode, ref 0 nidall nid={0@lo 192.168.201.54@tcp} 10, seq 2, type=nid, ref 0 default * 10000, seq 1, type=nid, ref 0 high_priority_requests: CPT 0: nidwr nid={0@lo 192.168.201.54@tcp}&opcode={ost_write} 15, seq 4, type=nid+opcode, ref 0 nidrd nid={0@lo 192.168.201.54@tcp}&opcode={ost_read} 5, seq 3, type=nid+opcode, ref 0 nidall nid={0@lo 192.168.201.54@tcp} 10, seq 2, type=nid, ref 0 default * 10000, seq 1, type=nid, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.87578 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055222 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116907 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100922 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224375 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196771 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126494 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143158 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212388 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214637 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292485 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235923 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271408 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339916 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394195 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176618 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352791 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665174 s, 15.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10661 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834996 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640955 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644223 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113887 s, 9.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760527 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790867 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836454 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10086 s, 10.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100122 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124863 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114567 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174667 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153285 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155653 s, 6.7 MB/s removed '/mnt/lustre/d77tb.sanityn/nrs_r_oleg154-client.virtnet' removed '/mnt/lustre/d77tb.sanityn/nrs_w_oleg154-client.virtnet' removed directory '/mnt/lustre/d77tb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96211 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.724 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77tb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nidall ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_tbf_rule=stop nidwr ost.OSS.ost_io.nrs_policies=fifo PASS 77tb (80s) == sanityn test 77tc: Testing for the setting of allowed TBF classification type ========================================================== 21:38:47 (1788831527) oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid classify=dynamic allow=nid allow=nid: File exists oleg154-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid classify=dynamic allow=nid allow=nid': File exists pdsh@oleg154-client: oleg154-server: ssh exited with exit code 17 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nid allow=nid allow=uid: Invalid argument oleg154-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nid allow=nid allow=uid': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic allow=nid+opcode allow=nid allow=uid+opcode oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start another uid={500} rate=10: Operation not permitted oleg154-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start another uid={500} rate=10': Operation not permitted pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start another uid={500}&opcode={ost_read}&gid={500} rate=10: Operation not permitted oleg154-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start another uid={500}&opcode={ost_read}&gid={500} rate=10': Operation not permitted pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 ost.OSS.ost_io.nrs_tbf_rule=start nidall nid={0@lo} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start nidrd nid={0@lo}&opcode={ost_read} rate=5 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop nidall ost.OSS.ost_io.nrs_tbf_rule=stop nidrd ost.OSS.ost_io.nrs_policies=fifo oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf nodemap classify=dynamic allow=nid allow=uid allow=gid allow=jobid allow=opcode allow=projid allow=nid+uid allow=nid+gid allow=uid+gid allow=nid+jobid allow=uid+jobid allow=gid+jobid allow=nid+opcode allow=uid+opcode allow=gid+opcode allow=nid+projid: Argument list too long oleg154-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf nodemap classify=dynamic allow=nid allow=uid allow=gid allow=jobid allow=opcode allow=projid allow=nid+uid allow=nid+gid allow=uid+gid allow=nid+jobid allow=uid+jobid allow=gid+jobid allow=nid+opcode allow=uid+opcode allow=gid+opcode allow=nid+projid': Argument list too long pdsh@oleg154-client: oleg154-server: ssh exited with exit code 7 Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo PASS 77tc (33s) == sanityn test 77td: Testing Number of allowed rules larger or smaller than threshold ========================================================== 21:39:20 (1788831560) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre opts: Stopping clients: oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg154-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg154-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg154-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg154-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.54,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg154-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.54,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg154-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.54,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69781/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.54,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg154-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: kmemleak disabled oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: MODOPTS_LIBCFS= oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre Starting client oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre Started clients oleg154-client.virtnet: 192.168.201.154@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre2 Starting client oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre2 Started clients oleg154-client.virtnet: 192.168.201.154@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88d8d2567000.idle_timeout=debug osc.lustre-OST0000-osc-ffff88d8ee47b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88d8d2567000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88d8ee47b800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic allow=projid+opcode allow=projid ost.OSS.ost_io.nrs_tbf_allowed_types= regular_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid high_priority_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid 8 ost.OSS.ost_io.nrs_tbf_rule=start rulerd0 projid={100}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr0 projid={100}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start ruleall0 projid={200} rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall0 projid={200} 15, seq 4, type=projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=nid+opcode+projid, ref 0 high_priority_requests: CPT 0: ruleall0 projid={200} 15, seq 4, type=projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=nid+opcode+projid, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.493443 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419427 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751374 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114076 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060553 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164679 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239006 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216197 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284721 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338404 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237957 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301934 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323988 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375459 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328962 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250544 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309448 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973688 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419033 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053028 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545648 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630245 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224468 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652096 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744602 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846895 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183814 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903142 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943601 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165896 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103349 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111364 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118287 s, 8.9 MB/s removed '/mnt/lustre/d77td.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77td.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77td.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.2362 s, 9.3 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6286 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.60349 s, 13.8 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.74425 s, 15.5 MB/s Read runtime is 8 s, speed is 12.500000 IOPS cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=change default rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall0 projid={200} 15, seq 4, type=projid, ref 1 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 1 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 1 default * 15, seq 1, type=nid+opcode+projid, ref 2 high_priority_requests: CPT 0: ruleall0 projid={200} 15, seq 4, type=projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 15, seq 1, type=nid+opcode+projid, ref 1 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.54484 s, 12.3 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.88979 s, 15.2 MB/s Read runtime is 8 s, speed is 12.500000 IOPS cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nid classify=dynamic allow=projid+opcode allow=projid ost.OSS.ost_io.nrs_tbf_allowed_types= regular_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid high_priority_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000002 nid 8 ost.OSS.ost_io.nrs_tbf_rule=start rulerd0 projid={100}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr0 projid={100}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd1 projid={101}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr1 projid={101}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd2 projid={102}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr2 projid={102}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd3 projid={103}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr3 projid={103}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rulerd4 projid={104}&opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start rulewr4 projid={104}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start ruleall0 projid={200} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall1 projid={201} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall2 projid={202} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall3 projid={203} rate=15 ost.OSS.ost_io.nrs_tbf_rule=start ruleall4 projid={204} rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall4 projid={204} 15, seq 16, type=projid, ref 0 ruleall3 projid={203} 15, seq 15, type=projid, ref 0 ruleall2 projid={202} 15, seq 14, type=projid, ref 0 ruleall1 projid={201} 15, seq 13, type=projid, ref 0 ruleall0 projid={200} 15, seq 12, type=projid, ref 0 rulewr4 projid={104}&opcode={ost_write} 10, seq 11, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, seq 10, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, seq 9, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, seq 8, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, seq 7, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, seq 6, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, seq 5, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, seq 4, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=nid+opcode+projid, ref 0 high_priority_requests: CPT 0: ruleall4 projid={204} 15, seq 16, type=projid, ref 0 ruleall3 projid={203} 15, seq 15, type=projid, ref 0 ruleall2 projid={202} 15, seq 14, type=projid, ref 0 ruleall1 projid={201} 15, seq 13, type=projid, ref 0 ruleall0 projid={200} 15, seq 12, type=projid, ref 0 rulewr4 projid={104}&opcode={ost_write} 10, seq 11, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, seq 10, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, seq 9, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, seq 8, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, seq 7, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, seq 6, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, seq 5, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, seq 4, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=nid+opcode+projid, ref 0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.759266 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699728 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837661 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746989 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141386 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241135 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127977 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182721 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277172 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318924 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18547 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322916 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204251 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105005 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320872 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88d8ee47b800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88d8ee47b800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294389 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328664 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237335 s, 44.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808828 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707156 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113345 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499128 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646949 s, 16.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106656 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947152 s, 11.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091016 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712715 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996956 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990157 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101199 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869394 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148897 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106934 s, 9.8 MB/s removed '/mnt/lustre/d77td.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77td.sanityn/nrs_r_oleg154-client.virtnet' removed directory '/mnt/lustre/d77td.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.2866 s, 9.3 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6531 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.6457 s, 9.0 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.97745 s, 15.0 MB/s Read runtime is 8 s, speed is 12.500000 IOPS cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=change default rate=15 ost.OSS.ost_io.nrs_tbf_rule= regular_requests: CPT 0: ruleall4 projid={204} 15, seq 16, type=projid, ref 0 ruleall3 projid={203} 15, seq 15, type=projid, ref 0 ruleall2 projid={202} 15, seq 14, type=projid, ref 0 ruleall1 projid={201} 15, seq 13, type=projid, ref 0 ruleall0 projid={200} 15, seq 12, type=projid, ref 1 rulewr4 projid={104}&opcode={ost_write} 10, seq 11, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, seq 10, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, seq 9, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, seq 8, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, seq 7, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, seq 6, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, seq 5, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, seq 4, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 1 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 1 default * 15, seq 1, type=nid+opcode+projid, ref 2 high_priority_requests: CPT 0: ruleall4 projid={204} 15, seq 16, type=projid, ref 0 ruleall3 projid={203} 15, seq 15, type=projid, ref 0 ruleall2 projid={202} 15, seq 14, type=projid, ref 0 ruleall1 projid={201} 15, seq 13, type=projid, ref 0 ruleall0 projid={200} 15, seq 12, type=projid, ref 0 rulewr4 projid={104}&opcode={ost_write} 10, seq 11, type=opcode+projid, ref 0 rulerd4 projid={104}&opcode={ost_read} 5, seq 10, type=opcode+projid, ref 0 rulewr3 projid={103}&opcode={ost_write} 10, seq 9, type=opcode+projid, ref 0 rulerd3 projid={103}&opcode={ost_read} 5, seq 8, type=opcode+projid, ref 0 rulewr2 projid={102}&opcode={ost_write} 10, seq 7, type=opcode+projid, ref 0 rulerd2 projid={102}&opcode={ost_read} 5, seq 6, type=opcode+projid, ref 0 rulewr1 projid={101}&opcode={ost_write} 10, seq 5, type=opcode+projid, ref 0 rulerd1 projid={101}&opcode={ost_read} 5, seq 4, type=opcode+projid, ref 0 rulewr0 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rulerd0 projid={100}&opcode={ost_read} 5, seq 2, type=opcode+projid, ref 0 default * 15, seq 1, type=nid+opcode+projid, ref 1 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 15, read rate: 15 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.43056 s, 12.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.74268 s, 15.6 MB/s Read runtime is 8 s, speed is 12.500000 IOPS cleanup_tbf /mnt/lustre/d77td.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo PASS 77td (423s) == sanityn test 77tf: TBF: rule churn under load on a dynamic classify policy ========================================================== 21:46:23 (1788831983) jobid_name=%e.%p jobid_var=TEST_77TF_JOBVAR ost.OSS.ost_io.nrs_policies=tbf jobid+opcode classify=dynamic ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=start churn1 jobid={dd.*} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=start churn2 jobid={dd.*}&opcode={ost_write} rate=2000 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_tbf_rule=stop churn1 Cleanup TBF ost.OSS.ost_io.nrs_tbf_rule=stop churn1 ost.OSS.ost_io.nrs_tbf_rule=stop churn2 ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77tf (65s) == sanityn test 77te: Dynamic type lookup should obey the rule order ========================================================== 21:47:28 (1788832048) using SAVE_PROJECT_SUPPORTED=0 ost.OSS.ost_io.nrs_policies=tbf projid classify=dynamic allow=projid+opcode allow=uid ost.OSS.ost_io.nrs_tbf_allowed_types= regular_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000008 uid high_priority_allowed_types: CPT 0: 0: 0x00000024 opcode+projid 1: 0x00000020 projid 2: 0x00000008 uid ost.OSS.ost_io.nrs_tbf_rule=start rule1 projid={100}&opcode={ost_read} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rule2 projid={100}&opcode={ost_write} rate=10 ost.OSS.ost_io.nrs_tbf_rule=start rule3 uid={500} rate=5 regular_requests: CPT 0: rule3 uid={500} 5, seq 4, type=uid, ref 0 rule2 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rule1 projid={100}&opcode={ost_read} 10, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=opcode+uid+projid, ref 0 high_priority_requests: CPT 0: rule3 uid={500} 5, seq 4, type=uid, ref 0 rule2 projid={100}&opcode={ost_write} 10, seq 3, type=opcode+projid, ref 0 rule1 projid={100}&opcode={ost_read} 10, seq 2, type=opcode+projid, ref 0 default * 10000, seq 1, type=opcode+uid+projid, ref 0 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9335 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0184 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77te.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77te.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7789 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77te.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9485 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77te.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_policies=fifo PASS 77te (129s) == sanityn test 77tg: TBF: the default rule holds its rate with many rules ========================================================== 21:49:37 (1788832177) ost.OSS.ost_io.nrs_policies=tbf uid classify=dynamic allow=uid ost.OSS.ost_io.nrs_tbf_rule=start r5000 uid={5000} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5001 uid={5001} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5002 uid={5002} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5003 uid={5003} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5004 uid={5004} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5005 uid={5005} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5006 uid={5006} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5007 uid={5007} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5008 uid={5008} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5009 uid={5009} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5010 uid={5010} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5011 uid={5011} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5012 uid={5012} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5013 uid={5013} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5014 uid={5014} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5015 uid={5015} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5016 uid={5016} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5017 uid={5017} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5018 uid={5018} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5019 uid={5019} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5020 uid={5020} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5021 uid={5021} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5022 uid={5022} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5023 uid={5023} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5024 uid={5024} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5025 uid={5025} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5026 uid={5026} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5027 uid={5027} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5028 uid={5028} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5029 uid={5029} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5030 uid={5030} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5031 uid={5031} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5032 uid={5032} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5033 uid={5033} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5034 uid={5034} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5035 uid={5035} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5036 uid={5036} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5037 uid={5037} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5038 uid={5038} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5039 uid={5039} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5040 uid={5040} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5041 uid={5041} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5042 uid={5042} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5043 uid={5043} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5044 uid={5044} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5045 uid={5045} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5046 uid={5046} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5047 uid={5047} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5048 uid={5048} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5049 uid={5049} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5050 uid={5050} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5051 uid={5051} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5052 uid={5052} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5053 uid={5053} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5054 uid={5054} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5055 uid={5055} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5056 uid={5056} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5057 uid={5057} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5058 uid={5058} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5059 uid={5059} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5060 uid={5060} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5061 uid={5061} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5062 uid={5062} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5063 uid={5063} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5064 uid={5064} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5065 uid={5065} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5066 uid={5066} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5067 uid={5067} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5068 uid={5068} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5069 uid={5069} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5070 uid={5070} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5071 uid={5071} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5072 uid={5072} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5073 uid={5073} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5074 uid={5074} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5075 uid={5075} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5076 uid={5076} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5077 uid={5077} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5078 uid={5078} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5079 uid={5079} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5080 uid={5080} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5081 uid={5081} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5082 uid={5082} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5083 uid={5083} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5084 uid={5084} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5085 uid={5085} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5086 uid={5086} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5087 uid={5087} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5088 uid={5088} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5089 uid={5089} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5090 uid={5090} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5091 uid={5091} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5092 uid={5092} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5093 uid={5093} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5094 uid={5094} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5095 uid={5095} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5096 uid={5096} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5097 uid={5097} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5098 uid={5098} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5099 uid={5099} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5100 uid={5100} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5101 uid={5101} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5102 uid={5102} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5103 uid={5103} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5104 uid={5104} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5105 uid={5105} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5106 uid={5106} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5107 uid={5107} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5108 uid={5108} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5109 uid={5109} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5110 uid={5110} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5111 uid={5111} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5112 uid={5112} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5113 uid={5113} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5114 uid={5114} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5115 uid={5115} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5116 uid={5116} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5117 uid={5117} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5118 uid={5118} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5119 uid={5119} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5120 uid={5120} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5121 uid={5121} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5122 uid={5122} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5123 uid={5123} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5124 uid={5124} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5125 uid={5125} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5126 uid={5126} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5127 uid={5127} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5128 uid={5128} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5129 uid={5129} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5130 uid={5130} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5131 uid={5131} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5132 uid={5132} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5133 uid={5133} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5134 uid={5134} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5135 uid={5135} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5136 uid={5136} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5137 uid={5137} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5138 uid={5138} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5139 uid={5139} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5140 uid={5140} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5141 uid={5141} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5142 uid={5142} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5143 uid={5143} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5144 uid={5144} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5145 uid={5145} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5146 uid={5146} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5147 uid={5147} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5148 uid={5148} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5149 uid={5149} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5150 uid={5150} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5151 uid={5151} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5152 uid={5152} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5153 uid={5153} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5154 uid={5154} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5155 uid={5155} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5156 uid={5156} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5157 uid={5157} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5158 uid={5158} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5159 uid={5159} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5160 uid={5160} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5161 uid={5161} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5162 uid={5162} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5163 uid={5163} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5164 uid={5164} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5165 uid={5165} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5166 uid={5166} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5167 uid={5167} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5168 uid={5168} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5169 uid={5169} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5170 uid={5170} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5171 uid={5171} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5172 uid={5172} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5173 uid={5173} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5174 uid={5174} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5175 uid={5175} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5176 uid={5176} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5177 uid={5177} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5178 uid={5178} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5179 uid={5179} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5180 uid={5180} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5181 uid={5181} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5182 uid={5182} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5183 uid={5183} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5184 uid={5184} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5185 uid={5185} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5186 uid={5186} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5187 uid={5187} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5188 uid={5188} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5189 uid={5189} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5190 uid={5190} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5191 uid={5191} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5192 uid={5192} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5193 uid={5193} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5194 uid={5194} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5195 uid={5195} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5196 uid={5196} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5197 uid={5197} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5198 uid={5198} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start r5199 uid={5199} rate=1000 not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5383 s, 10.0 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99364 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77tg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Cleanup TBF ost.OSS.ost_io.nrs_policies=fifo PASS 77tg (66s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 21:50:43 (1788832243) oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg154-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg154-client: oleg154-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (10s) == sanityn test 79: xattr: intent error ================== 21:50:54 (1788832254) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115272381366410 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 21:50:56.000000000 -0400 Modify: 2026-09-07 21:50:56.000000000 -0400 Change: 2026-09-07 21:50:56.000000000 -0400 Birth: 2026-09-07 21:50:56.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (21s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 21:51:14 (1788832274) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (4s) == sanityn test 80b: Accessing directory during migration ========================================================== 21:51:18 (1788832278) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 21:51:22 (1788832282) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 21:51:25 (1788832285) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (4s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 21:51:30 (1788832290) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 21:51:33 (1788832293) - open/close 716 (time 1788832310.38 total 10.01 last 71.53) - open/close 1429 (time 1788832320.38 total 20.02 last 71.23) - open/close 2225 (time 1788832330.40 total 30.03 last 79.50) total: 2500 open/close in 33.43 seconds: 74.78 ops/second snapshot_time 1788832435.035219965 secs.nsecs start_time 1788832297.646439743 secs.nsecs elapsed_time 137.388780222 secs.nsecs open 2505 samples [usecs] 132 27848 3572412 8431473678 close 2504 samples [usecs] 165 11660 935635 837933519 mknod 2501 samples [usecs] 591 10598 3141602 6156328848 unlink 2500 samples [usecs] 669 57639 4989672 21617988682 mkdir 2 samples [usecs] 1039 1296 2335 2759137 rmdir 2 samples [usecs] 12123 15296 27419 380934745 rename 2500 samples [usecs] 492 80193 18295384 277187634214 getattr 3760 samples [usecs] 10 9938 287166 309343976 parallel_rename_file 2500 samples [usecs] 492 80193 18295384 277187634214 crossdir_rename 2500 samples [usecs] 492 80193 18295384 277187634214 readpage 5 samples [usecs] 5916 31835 77518 1622301732 PASS 81d (147s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 21:54:01 (1788832441) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7614 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7614 PASS 82 (9s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 21:54:09 (1788832449) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 21:54:13 (1788832453) fail_loc=0x8000060b PASS 84 (14s) == sanityn test 85: Lustre API root cache race =========== 21:54:27 (1788832467) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788832469 Finishing test test1 at 1788832475 PASS 85 (16s) == sanityn test 90: open/create and unlink striped directory ========================================================== 21:54:43 (1788832483) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (4s) == sanityn test 91: chmod and unlink striped directory === 21:54:47 (1788832487) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 21:54:52 (1788832492) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 21:54:56 (1788832496) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5770 0x168a 0x280000400 0 5830 0x16c6 0x240000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5831 0x16c7 0x240000400 1 5771 0x168b 0x280000400 rc1=0 and rc2=0 PASS 93 (21s) == sanityn test 94: signal vs CP callback race =========== 21:55:17 (1788832517) osc.lustre-OST0000-osc-ffff88d8d2567000.idle_timeout=0 osc.lustre-OST0000-osc-ffff88d8ee47b800.idle_timeout=0 osc.lustre-OST0001-osc-ffff88d8d2567000.idle_timeout=0 osc.lustre-OST0001-osc-ffff88d8ee47b800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.33392 s, 24.5 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.0043 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 7024: 309512 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0700099 s, 58.5 kB/s osc.lustre-OST0000-osc-ffff88d8d2567000.idle_timeout=debug osc.lustre-OST0000-osc-ffff88d8ee47b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88d8d2567000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88d8ee47b800.idle_timeout=debug PASS 94 (15s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 21:55:32 (1788832532) llite.lustre-ffff88d8d2567000.fast_read=0 llite.lustre-ffff88d8ee47b800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0814198 s, 25.8 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff88d8d2567000.fast_read=1 llite.lustre-ffff88d8ee47b800.fast_read=1 PASS 95a (20s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 21:55:52 (1788832552) llite.lustre-ffff88d8d2567000.fast_read=0 llite.lustre-ffff88d8ee47b800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00975739 s, 1.3 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff88d8d2567000.fast_read=1 llite.lustre-ffff88d8ee47b800.fast_read=1 PASS 95b (15s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 21:56:07 (1788832567) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 21:56:11 (1788832571) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00788115 s, 520 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (9s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 21:56:20 (1788832580) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.235739 s, 8.9 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (10s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 21:56:30 (1788832590) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.204721 s, 10.2 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (9s) == sanityn test 100e: DoM: read on open and file size ==== 21:56:39 (1788832599) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0115227 s, 1.0 MB/s PASS 100e (9s) == sanityn test 101a: Discard DoM data on unlink ========= 21:56:48 (1788832608) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0108409 s, 378 kB/s PASS 101a (8s) == sanityn test 101b: Discard DoM data on rename ========= 21:56:56 (1788832616) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0184191 s, 222 kB/s PASS 101b (8s) == sanityn test 101c: Discard DoM data on close-unlink === 21:57:04 (1788832624) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0277701 s, 147 kB/s PASS 101c (10s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 21:57:14 (1788832634) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: a1 13 00 00 02 00 00 00 6b 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x2000013a1:0xa6b:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (9s) == sanityn test 103: Test size correctness with lockahead ========================================================== 21:57:23 (1788832643) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788832647 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788832651 Finishing test test23 at 1788832651 PASS 103 (18s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 21:57:41 (1788832661) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (5s) == sanityn test 105: Glimpse and lock cancel race ======== 21:57:46 (1788832666) fail_loc=0x416 PASS 105 (26s) == sanityn test 106a: Verify the btime via statx() ======= 21:58:12 (1788832692) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 21:58:17 (1788832697) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.016689 s, 61.4 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381368946 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 21:58:18.000000000 -0400 Modify: 2026-09-07 21:58:18.000000000 -0400 Change: 2026-09-07 21:58:18.000000000 -0400 Birth: 2026-09-07 21:58:19.000000000 -0400 /mnt/lustre/f106b.sanityn 144115272381368946 -rw-r--r-- 1 0 0 1788832699 1788832698 1788832698 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381368946 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 21:58:18.000000000 -0400 Modify: 2026-09-07 21:58:18.000000000 -0400 Change: 2026-09-07 21:58:18.000000000 -0400 Birth: 2026-09-07 21:58:19.000000000 -0400 1788832698 1024 1 PASS 106b (8s) == sanityn test 106c: Verify statx attributes mask ======= 21:58:25 (1788832705) PASS 106c (9s) == sanityn test 107a: Basic grouplock conflict =========== 21:58:35 (1788832715) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.299196 s, 35.0 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5840 0x16d0 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 First grouplock blocks second one PASS 107a (10s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 21:58:45 (1788832725) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.471258 s, 22.3 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5778 0x1692 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (15s) == sanityn test 108a: lseek: parallel updates ============ 21:59:00 (1788832740) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0242891 s, 412 kB/s llite.lustre-ffff88d8d2567000.hybrid_io=0 llite.lustre-ffff88d8ee47b800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.74957 s, 1.8 MB/s PASS 108a (13s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 21:59:14 (1788832754) 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre2 Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: kmemleak disabled oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: MODOPTS_LIBCFS= oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre2 PASS 109 (46s) == sanityn test 110: do not grant another lock on resend ========================================================== 21:59:59 (1788832799) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272414920707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 22:00:01.000000000 -0400 Modify: 2026-09-07 22:00:01.000000000 -0400 Change: 2026-09-07 22:00:01.000000000 -0400 Birth: 2026-09-07 22:00:02.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg154-client.virtnet oleg154-client.virtnet PASS 110 (169s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 22:02:48 (1788832968) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (3s) == sanityn test 112: update max-inherit in default LMV === 22:02:52 (1788832972) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 22:02:56 (1788832976) lustre-OST0000-osc-ffff88d8f4cb9000 active. lustre-OST0001-osc-ffff88d8f4cb9000 active. lustre-OST0000-osc-ffff88d8f4cb9000 active. lustre-OST0001-osc-ffff88d8f4cb9000 active. MGC192.168.201.154@tcp active. PASS 113 (9s) == sanityn test 114: implicit default LMV inherit ======== 22:03:05 (1788832985) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (4s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 22:03:09 (1788832989) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (5s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 22:03:14 (1788832994) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (3s) == sanityn test 121: trunc append race =================== 22:03:18 (1788832998) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0118138 s, 1.7 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.100524 s, 204 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (12s) == sanityn test 122: directory size is consistent across mounts ========================================================== 22:03:29 (1788833009) PASS 122 (12s) == sanityn test 200: service remains healthy while able to process request ========================================================== 22:03:41 (1788833021) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=12 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.152 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.23 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1949 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2994 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2672 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3686 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3541 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3215 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2943 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0518 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2402 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2817 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3141 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2383 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2635 s, 0.1 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2841 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.4496 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.4557 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1697 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3846 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4207 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4476 s, 0.1 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4019 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.6118 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg154-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88d8c8bfc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88d8c8bfc000.ost_server_uuid in IDLE state after 0 sec PASS 200 (89s) cleanup: ====================================================== == sanityn test complete, duration 10243 sec ============= 22:05:13 (1788833113) === sanityn: start cleanup 22:05:15 (1788833115) === Stopping clients: oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 22:10:02 (1788833402) ===