-----============= acceptance-small: sanityn ============----- Wed May 28 17:50:02 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg357-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanityn.*ex: No such file or directory excepting tests: 27 28 40a skipping tests SLOW=no: 33a === sanityn: start setup 17:50:07 (1748469007) === Starting client oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 Started clients oleg357-client.virtnet: 192.168.203.157@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg357-client.virtnet: executing check_config_client /mnt/lustre oleg357-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg357-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800abd01000.idle_timeout=debug osc.lustre-OST0000-osc-ffff88012b480000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800abd01000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012b480000.idle_timeout=debug disable quota as required oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanityn: finish setup 17:50:16 (1748469016) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106401 s, 98.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7921] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 17:50:17 (1748469017) PASS 0 (2s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 17:50:19 (1748469019) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (1s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 17:50:20 (1748469020) -rw-r--r-- 1 root root 0 May 28 17:50 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (2s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 17:50:22 (1748469022) -rw-r--r-- 1 root root 0 May 28 17:50 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (1s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 17:50:23 (1748469023) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (2s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 17:50:25 (1748469025) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (1s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 17:50:26 (1748469026) total 0 -rwxrwxrwx 1 root root 0 May 28 17:50 f2a -rwxrwxrwx 1 root root 0 May 28 17:50 f2b total 0 -rwxrwxrwx 1 root root 0 May 28 17:50 f2a -rwxrwxrwx 1 root root 0 May 28 17:50 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.00271046 s, 189 kB/s PASS 2e (2s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 17:50:28 (1748469028) /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has type file OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has perms 0777 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by user #500 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by group #500 OK Can't lstat /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn: No such file or directory PASS 2f (1s) == sanityn test 2g: check blocks update on sync write ==== 17:50:29 (1748469029) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0478499 s, 43.8 MB/s /mnt/lustre/f2g.sanityn has 4096 blocks /mnt/lustre2/f2g.sanityn has 4096 blocks PASS 2g (2s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 17:50:31 (1748469031) PASS 3 (1s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 17:50:32 (1748469032) PASS 4 (3s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 17:50:35 (1748469035) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (1s) == sanityn test 6: remove of open file on other node ============================================================================ 17:50:37 (1748469037) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (2s) == sanityn test 7: remove of open directory on other node ======================================================================= 17:50:38 (1748469038) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (1s) == sanityn test 8: remove of open special file on other node ==================================================================== 17:50:39 (1748469039) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (2s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 17:50:41 (1748469041) PASS 9a (1s) == sanityn test 9b: append to striped sparse file ======== 17:50:42 (1748469042) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101946 s, 103 MB/s 3+0 records in 3+0 records out 3 bytes (3 B) copied, 0.0093832 s, 0.3 kB/s Data read (expecting 'foo'): foo PASS 9b (2s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 17:50:44 (1748469044) 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00133023 s, 0.8 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00270702 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00247506 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00386105 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.0022487 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00228218 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00229253 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00322249 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00246227 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00264627 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00235185 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00258161 s, 0.4 kB/s PASS 10a (1s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 17:50:45 (1748469045) 1+0 records in 1+0 records out 3072 bytes (3.1 kB) copied, 0.00241889 s, 1.3 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0079067 s, 518 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB) copied, 0.000183107 s, 16.8 MB/s PASS 10b (2s) == sanityn test 11: execution of file opened for write should return error ============================================================== 17:50:47 (1748469047) striped dir -i1 -c2 -H crush /mnt/lustre/d11 multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7921 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 301: /mnt/lustre2/d11/f: Text file busy PASS 11 (1s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 17:50:48 (1748469048) warning: '-runtime' deprecated, use '-t runtime' instead - link 2947 (time 1748469059.45 total 10.00 last 294.69) - link 5805 (time 1748469069.46 total 20.00 last 285.76) - link 8614 (time 1748469079.46 total 30.00 last 280.81) - link 10000 (time 1748469084.26 total 34.81 last 288.70) - link 12840 (time 1748469094.26 total 44.81 last 283.93) - link 15466 (time 1748469104.26 total 54.81 last 262.54) - link 18686 (time 1748469114.27 total 64.81 last 321.96) - link 20000 (time 1748469118.44 total 68.99 last 314.49) - link 23093 (time 1748469128.44 total 78.99 last 309.30) - link 26445 (time 1748469138.45 total 88.99 last 335.15) - link 29866 (time 1748469148.45 total 98.99 last 342.03) total: 30193 link in 100.00 seconds: 301.93 ops/second using seed 566264872 running for 100 seconds - stat 10000 (time 1748469053 ; total 3 ; last 3) - stat 20000 (time 1748469056 ; total 6 ; last 3) - stat 30000 (time 1748469059 ; total 9 ; last 3) - stat 40000 (time 1748469062 ; total 12 ; last 3) - stat 50000 (time 1748469064 ; total 14 ; last 2) - stat 60000 (time 1748469067 ; total 17 ; last 3) - stat 70000 (time 1748469070 ; total 20 ; last 3) - stat 80000 (time 1748469073 ; total 23 ; last 3) - stat 90000 (time 1748469076 ; total 26 ; last 3) - stat 100000 (time 1748469079 ; total 29 ; last 3) - stat 110000 (time 1748469082 ; total 32 ; last 3) - stat 120000 (time 1748469085 ; total 35 ; last 3) - stat 130000 (time 1748469088 ; total 38 ; last 3) - stat 140000 (time 1748469091 ; total 41 ; last 3) - stat 150000 (time 1748469094 ; total 44 ; last 3) - stat 160000 (time 1748469097 ; total 47 ; last 3) - stat 170000 (time 1748469100 ; total 50 ; last 3) - stat 180000 (time 1748469108 ; total 58 ; last 8) - stat 190000 (time 1748469116 ; total 66 ; last 8) - stat 200000 (time 1748469124 ; total 74 ; last 8) - stat 210000 (time 1748469132 ; total 82 ; last 8) - stat 220000 (time 1748469139 ; total 89 ; last 7) - stat 230000 (time 1748469147 ; total 97 ; last 8) total: 233618 stats in 100 seconds: 2336.179932 stats/second - unlinked 0 (time 1748469100 ; total 0 ; last 0) - unlinked 10000 (time 1748469146 ; total 46 ; last 46) - unlinked 20000 (time 1748469171 ; total 71 ; last 25) - unlinked 30000 (time 1748469194 ; total 94 ; last 23) unlink(/mnt/lustre2/lockdir/lockfile30193) error: No such file or directory total: 30192 unlinks in 94 seconds: 321.191498 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (23630) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (23633) - No such process PASS 12 (147s) == sanityn test 13: test directory page revocation ======= 17:53:15 (1748469195) striped dir -i1 -c2 -H crush /mnt/lustre/d13 f13 f13 PASS 13 (2s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 17:53:17 (1748469197) striped dir -i0 -c2 -H crush2 /mnt/lustre/d14aa.sanityn multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7921 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 336: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 25050 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (2s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 17:53:19 (1748469199) striped dir -i0 -c2 -H all_char /mnt/lustre/d14ab.sanityn open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 25695 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (1s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 17:53:20 (1748469200) striped dir -i0 -c2 -H crush /mnt/lustre/d14b.sanityn truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 351: 26339 Terminated $DIR1/$tdir/sleep 60 PASS 14b (2s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 17:53:22 (1748469202) striped dir -i0 -c2 -H all_char /mnt/lustre/d14c.sanityn cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 363: 26986 Terminated $DIR1/$tdir/sleep 60 PASS 14c (1s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 17:53:23 (1748469203) striped dir -i0 -c2 -H all_char /mnt/lustre/d14d.sanityn chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 374: 27632 Terminated $DIR1/$tdir/sleep 60 PASS 14d (2s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 17:53:25 (1748469205) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_CODE=117440512 CLIENT_OS_VERSION_ID=7 Waiting for MDT destroys to complete 0 STRIPECOUNT=2 ORIGFREE=7207492 MAXFREE=2097152000 0 lustre-OST0000 avl=1824 grnt=1179 diff=645 limit=46331 FULL lustre-OST0001 avl=340 grnt=315 diff=25 limit=46331 FULL ERROR: dd blocks 7198273 != file size 7198292 LOG LOG2 file /tmp/oos2.log:dd: error writing '/mnt/lustre/oosfile': No space left on device /tmp/oos2.log:3605266+0 records in /tmp/oos2.log:3605265+0 records out /tmp/oos2.log:3691791360 bytes (3.7 GB) copied, 163.968 s, 22.5 MB/s /tmp/oos2.log2:dd: error writing '/mnt/lustre2/oosfile2': No space left on device /tmp/oos2.log2:3593009+0 records in /tmp/oos2.log2:3593008+0 records out /tmp/oos2.log2:3679240192 bytes (3.7 GB) copied, 163.936 s, 22.4 MB/s Waiting for MDT destroys to complete Waiting for MDT destroys to complete PASS 15 (196s) == sanityn test 16a: 2500 iterations of dual-mount fsx === 17:56:41 (1748469401) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.0817233 s, 103 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.0935281 s, 89.7 MB/s Chance of close/open is 1 in 50 Seed set to 9402 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn skipping zero length zero range truncating to largest ever: 0x2d061b fallocating to largest ever: 0x78759f fallocating to largest ever: 0x92e6e5 truncating to largest ever: 0x9af614 000100[0] 1748469403.591868 punch from 0x1663e7 to 0x170be8 (0xa802 bytes) zero_range to largest ever: 0x9dc265 000200[1] 1748469404.225744 read 0x258a2c thru 0x25b4f0 (0x2ac5 bytes) fallocating to largest ever: 0x9dfdd8 000300[1] 1748469404.916384 write 0x442324 thru 0x44f916 (0xd5f3 bytes) 000400[0] 1748469405.695356 trunc from 0x10cf41 to 0x11423d (0x72fd bytes) truncating to largest ever: 0x9efd1a 000500[0] 1748469406.312231 trunc from 0x9efd1a to 0x7022b2 (0x2eda67 bytes) 000600[0] 1748469407.012201 write 0x25f94a thru 0x2639e1 (0x4098 bytes) 000700[0] 1748469407.663343 write 0x572e4c thru 0x57b174 (0x8329 bytes) 000800[1] 1748469408.319140 mapread 0x800184 thru 0x8038d5 (0x3752 bytes) 000900[0] 1748469408.937882 fallocate 0x011031 thru 0x012d5a (0x1d2a bytes) 001000[1] 1748469409.738930 fallocate 0x1037ab thru 0x10ec81 (0xb4d7 bytes) 001100[0] 1748469410.358378 punch from 0x12e4e7 to 0x13e084 (0xfb9e bytes) 001200[1] 1748469411.030090 trunc from 0x030598 to 0x829dbb (0x7f9824 bytes) 001300[1] 1748469411.668520 trunc from 0x111699 to 0x11cc6d (0xb5d5 bytes) 001400[0] 1748469412.385786 trunc from 0x95fd3f to 0x87cc73 (0xe30cb bytes) 001500[1] 1748469413.053356 trunc from 0x6a8c4b to 0x6ad3b0 (0x4766 bytes) 001600[0] 1748469413.705851 read 0x609945 thru 0x60cd89 (0x3445 bytes) 001700[0] 1748469414.375341 mapread 0x79a47c thru 0x79ff59 (0x5ade bytes) 001800[1] 1748469415.080592 mapread 0x1bd284 thru 0x1c5a18 (0x8795 bytes) 002000[0] 1748469416.476626 punch from 0x24f835 to 0x25c4df (0xccab bytes) fallocating to largest ever: 0x9f52fb 002200[1] 1748469417.748525 punch from 0x359621 to 0x3656b9 (0xc099 bytes) truncating to largest ever: 0x9f7d6f 002400[0] 1748469419.001711 mapread 0x51155e thru 0x520c6d (0xf710 bytes) 002500[0] 1748469419.675661 trunc from 0xa00000 to 0x56e6a9 (0x491956 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 9419 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn truncating to largest ever: 0x3358ef truncating to largest ever: 0x68a7dd truncating to largest ever: 0x7aabd3 fallocating to largest ever: 0x88d8ce 000100[0] 1748469420.421227 trunc from 0x6151ff to 0x62014b (0xaf4d bytes) zero_range to largest ever: 0x8f0edc truncating to largest ever: 0x957cbf 000300[0] 1748469421.499622 write_OD 0x859000 thru 0x85dfff (0x5000 bytes) truncating to largest ever: 0x96060f truncating to largest ever: 0x9dbdfb 000400[1] 1748469421.961798 mapwrite 0x704000 thru 0x70f265 (0xb266 bytes) 000600[0] 1748469423.212959 trunc from 0x96fdfb to 0x3e1e8e (0x58df6c bytes) 000700[0] 1748469423.863467 punch from 0x1b36f7 to 0x1b8830 (0x513a bytes) 000800[0] 1748469424.393716 punch from 0x50f27a to 0x51c969 (0xd6f0 bytes) truncating to largest ever: 0x9f663f 000900[0] 1748469424.949540 mapwrite 0x7e3000 thru 0x7ece15 (0x9e16 bytes) 001000[1] 1748469425.473495 mapread 0x422000 thru 0x42e8e3 (0xc8e4 bytes) 001100[1] 1748469425.949739 punch from 0x750a2c to 0x75ee5c (0xe431 bytes) 001300[0] 1748469427.083039 write 0x3ed000 thru 0x3fca52 (0xfa53 bytes) 001400[1] 1748469427.617356 fallocate 0x5506e8 thru 0x553c0a (0x3523 bytes) 001500[1] 1748469428.234389 trunc from 0x0a0e46 to 0x0a4e94 (0x404f bytes) 001700[0] 1748469429.265808 write 0x213000 thru 0x21c618 (0x9619 bytes) 001900[0] 1748469430.349717 mapread 0x0b4000 thru 0x0b5508 (0x1509 bytes) 002000[1] 1748469431.011692 trunc from 0x4ca837 to 0x4d55aa (0xad74 bytes) 002200[1] 1748469432.169052 punch from 0x3a595d to 0x3ab0db (0x577f bytes) 002300[1] 1748469432.668360 mapread 0x449000 thru 0x4494ff (0x500 bytes) 002400[0] 1748469433.187375 read_OD 0x14d000 thru 0x14dfff (0x1000 bytes) All operations completed A-OK! PASS 16a (33s) == sanityn test 16b: 2500 iterations of dual-mount fsx at small size ========================================================== 17:57:14 (1748469434) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.0971578 s, 86.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.0937092 s, 89.5 MB/s Chance of close/open is 1 in 50 Seed set to 9435 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn truncating to largest ever: 0x254b5 fallocating to largest ever: 0x2e5cfb truncating to largest ever: 0x8d41b1 truncating to largest ever: 0x9e7f47 truncating to largest ever: 0x9f5ee6 000100[0] 1748469436.244768 fallocate 0x586df8 thru 0x588528 (0x1731 bytes) 000200[1] 1748469436.877984 trunc from 0x1267a9 to 0x1273d2 (0xc2a bytes) 000300[0] 1748469437.424702 fallocate 0x687a75 thru 0x68802e (0x5ba bytes) 000500[1] 1748469438.501616 punch from 0x21512e to 0x216090 (0xf63 bytes) 000700[0] 1748469439.640284 mapwrite 0x2f80bf thru 0x2f93ee (0x1330 bytes) fallocating to largest ever: 0x9fd65f 000900[1] 1748469440.994087 read 0x30ca3e thru 0x30d84e (0xe11 bytes) 001000[1] 1748469441.529116 write 0x83155c thru 0x831ec4 (0x969 bytes) 001100[1] 1748469442.031627 punch from 0x47d2d3 to 0x47f027 (0x1d55 bytes) 001200[0] 1748469442.659558 mapwrite 0x1c5091 thru 0x1c606a (0xfda bytes) 001300[1] 1748469443.248506 punch from 0x86f0d3 to 0x87037e (0x12ac bytes) 001400[0] 1748469443.885478 punch from 0x060c4e to 0x0621b9 (0x156c bytes) 001500[1] 1748469444.480229 mapwrite 0x731728 thru 0x731b89 (0x462 bytes) 001600[1] 1748469445.030357 punch from 0x694a96 to 0x695b74 (0x10df bytes) 001800[0] 1748469446.151225 trunc from 0x7cb1a5 to 0x5cc0ed (0x1ff0b7 bytes) 001900[0] 1748469446.780512 mapread 0x2c873c thru 0x2c8d1c (0x5e1 bytes) 002000[1] 1748469447.317272 trunc from 0x0d72bc to 0x0d7969 (0x6ae bytes) 002100[1] 1748469447.810528 punch from 0x67da30 to 0x67dbc2 (0x193 bytes) 002200[1] 1748469448.323210 trunc from 0x2e0213 to 0x2e17eb (0x15d9 bytes) 002300[1] 1748469448.855325 trunc from 0x8f4e51 to 0x421482 (0x4d39ce bytes) 002400[0] 1748469449.417132 read 0x019dc7 thru 0x01a6f3 (0x92d bytes) 002500[0] 1748469450.017689 write 0x8a1fd1 thru 0x8a202a (0x5a bytes) All operations completed A-OK! PASS 16b (17s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 17:57:31 (1748469451) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.102323 s, 82.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.0888657 s, 94.4 MB/s Waiting for MDT destroys to complete Chance of close/open is 1 in 50 Seed set to 9460 fd 0: /mnt/lustre/f16c.sanityn fd 1: /mnt/lustre2/f16c.sanityn skipping zero length zero range skipping zero length punch hole skipping zero length zero range skipping zero size read skipping zero length punch hole skipping zero length zero range skipping zero size read skipping zero length zero range fallocating to largest ever: 0x50272f skipping zero size read skipping zero size read truncating to largest ever: 0x7d8c1f truncating to largest ever: 0x96d47f truncating to largest ever: 0x9d3ed6 000100[1] 1748469461.132327 fallocate 0x5d90db thru 0x5e6dcd (0xdcf3 bytes) truncating to largest ever: 0x9f235e 000200[0] 1748469461.919105 read 0x413538 thru 0x4172ee (0x3db7 bytes) 000300[0] 1748469462.569013 mapwrite 0x711b6d thru 0x71bb14 (0x9fa8 bytes) 000400[0] 1748469463.288611 write 0x38c089 thru 0x398360 (0xc2d8 bytes) 000500[1] 1748469463.995053 trunc from 0x5ab2c7 to 0x6caee3 (0x11fc1d bytes) 000600[0] 1748469464.554441 read 0x13b911 thru 0x14aab5 (0xf1a5 bytes) 000700[0] 1748469465.176098 read 0x148b19 thru 0x149cf2 (0x11da bytes) 000800[0] 1748469465.804911 trunc from 0x182007 to 0x188eff (0x6ef9 bytes) 000900[0] 1748469466.420895 read 0x2e3d10 thru 0x2eba86 (0x7d77 bytes) 001000[0] 1748469467.041597 mapread 0x2e489a thru 0x2f4378 (0xfadf bytes) 001100[0] 1748469467.702805 punch from 0x11a167 to 0x120c53 (0x6aed bytes) 001200[0] 1748469468.390425 punch from 0x12db40 to 0x1366a7 (0x8b68 bytes) 001300[1] 1748469469.054473 trunc from 0x57d167 to 0x65f51c (0xe23b6 bytes) 001400[1] 1748469469.709062 write 0x2f3fd1 thru 0x2f4e89 (0xeb9 bytes) fallocating to largest ever: 0xa00000 001600[0] 1748469471.017735 write 0x1e9ff6 thru 0x1f2ec8 (0x8ed3 bytes) 001700[1] 1748469471.612087 punch from 0x80a3c6 to 0x80f379 (0x4fb4 bytes) 001800[1] 1748469472.343232 fallocate 0x613553 thru 0x615f33 (0x29e1 bytes) 001900[1] 1748469473.020398 read 0x7276ab thru 0x734a8a (0xd3e0 bytes) 002000[1] 1748469473.639226 mapread 0x4230c5 thru 0x423321 (0x25d bytes) 002100[1] 1748469474.330074 write 0x74d498 thru 0x759176 (0xbcdf bytes) 002200[0] 1748469474.980696 trunc from 0x856818 to 0x00df60 (0x8488b7 bytes) 002300[1] 1748469475.681044 read 0x4c49e5 thru 0x4d033e (0xb95a bytes) 002400[0] 1748469476.227155 mapread 0x4865b1 thru 0x4910a0 (0xaaf0 bytes) 002500[0] 1748469476.990684 fallocate 0x57f47b thru 0x58145b (0x1fe1 bytes) All operations completed A-OK! PASS 16c (27s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 17:57:58 (1748469478) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.0915642 s, 91.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.0982423 s, 85.4 MB/s ldlm.namespaces.MGC192.168.203.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800abd01000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff88012b480000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800abd01000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff88012b480000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800abd01000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800abd01000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 1.57019 s, 66.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.625218 s, 168 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.637913 s, 164 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 1.17307 s, 89.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.727982 s, 144 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.723522 s, 145 MB/s PASS 16d (8s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 17:58:06 (1748469486) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0604843 s, 173 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113863 s, 92.1 MB/s PASS 16e (2s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 17:58:08 (1748469488) TIMEOUT PASS 16f (22s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 17:58:30 (1748469510) TIMEOUT PASS 16g (21s) == sanityn test 16h: mmap read after truncate file ======= 17:58:51 (1748469531) striped dir -i0 -c2 -H all_char /mnt/lustre/d16h.sanityn ------- before truncate -------- 0003220 3035 3a31 3035 3a31 2f3a 6f68 656d 732f 0003240 6e61 7469 7579 7273 3a31 622f 6e69 622f 0003260 7361 0a68 0003264 ------- after truncate -------- 0001460 7373 6468 783a 373a 3a34 3437 503a 6972 0001500 6976 656c 6567 732d 7065 7261 7461 6465 0001520 5320 4853 2f3a 6176 2f72 0001532 PASS 16h (2s) == sanityn test 16i: read after truncate file ============ 17:58:53 (1748469533) striped dir -i0 -c2 -H crush2 /mnt/lustre/d16i.sanityn ------- before truncate -------- 0003220 3035 3a31 3035 3a31 2f3a 6f68 656d 732f 0003240 6e61 7469 7579 7273 3a31 622f 6e69 622f 0003260 7361 0a68 0003264 ------- after truncate -------- 0001460 7373 6468 783a 373a 3a34 3437 503a 6972 0001500 6976 656c 6567 732d 7065 7261 7461 6465 0001520 5320 4853 2f3a 6176 2f72 0001532 PASS 16i (1s) == sanityn test 16j: race dio with buffered i/o ========== 17:58:54 (1748469534) 1+0 records in 1+0 records out 26214400 bytes (26 MB) copied, 0.347605 s, 75.4 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.260034 s, 101 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.306998 s, 85.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.165195 s, 159 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.252703 s, 104 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.170613 s, 154 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.309793 s, 84.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.276592 s, 94.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.30109 s, 87.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.331569 s, 79.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.409855 s, 64.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.332651 s, 78.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.345541 s, 75.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.358334 s, 73.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.446071 s, 58.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.31433 s, 83.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.338304 s, 77.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.354351 s, 74.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.40996 s, 63.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (7s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 17:59:01 (1748469541) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 skipping zero size read truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 skipping zero size read truncating to largest ever: 0x3c569db fallocating to largest ever: 0x3428364 skipping zero size read truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Wed May 28 17:59:05 EDT 2025 All operations completed A-OK! All operations completed A-OK! + fsx 2 done: rc=0 Wed May 28 17:59:06 EDT 2025 + fsx 3 done: rc=0 Wed May 28 17:59:06 EDT 2025 + fsx 4 done: rc=0 Wed May 28 17:59:06 EDT 2025 All operations completed A-OK! + fsx 5 done: rc=0 Wed May 28 17:59:06 EDT 2025 PASS 16k (6s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 17:59:07 (1748469547) fail_loc=0x8000030a PASS 17 (4s) == sanityn test 18: mmap sanity check =========================================================================================== 17:59:11 (1748469551) mmap test1: basic mmap operation (PASS, 0.018643s) mmap test2: MAP_PRIVATE not write back (PASS, 0.013566s) mmap test3: concurrent mmap ops on two nodes (PASS, 22.713s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.07s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.02652s) mmap test6: check mmap write/read content on two nodes (PASS, 0.028467s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.33396s) mmap test11: multi-thread shared mmap (PASS, 0.30177s) mmap test10: mtime not change for readonly mmap access (PASS, 1.017s) PASS 18 (31s) == sanityn test 19: test concurrent uncached read races ========================================================================= 17:59:42 (1748469582) 32+0 records in 32+0 records out 16777216 bytes (17 MB) copied, 0.140995 s, 119 MB/s loop 5 loop 10 loop 15 loop 20 PASS 19 (8s) == sanityn test 20: test extra readahead page left in cache ============================================================== 17:59:50 (1748469590) striped dir -i0 -c2 -H crush /mnt/lustre/d20.sanityn PASS 20 (1s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 17:59:51 (1748469591) striped dir -i1 -c2 -H crush /mnt/lustre/d21.sanityn rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (2s) == sanityn test 23: others should see updated atime while another read============================================================== 17:59:53 (1748469593) now is 1748469594 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7921 new atime is 1748469655 PASS 23 (63s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 18:00:56 (1748469656) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 30676 3576344 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 8256 3598764 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38932 7175108 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2056 1285632 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 30676 3576344 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 8256 3598764 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 38932 7175108 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 350 999.7K 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1000.0K 279 999.7K 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 256.0K 402 255.6K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 256.0K 371 255.6K 1% /mnt/lustre[OST:1] filesystem_summary: 511.9K 629 511.2K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 350 999.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1000.0K 279 999.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 256.0K 402 255.6K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 256.0K 371 255.6K 1% /mnt/lustre2[OST:1] filesystem_summary: 511.9K 629 511.2K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 1.3G 2.0M 1.2G 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1.3G 1.8M 1.2G 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3.7G 30.0M 3.4G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.7G 8.1M 3.4G 1% /mnt/lustre[OST:1] filesystem_summary: 7.3G 38.0M 6.8G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 350 1023650 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1024000 279 1023721 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 262144 402 261742 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 262144 371 261773 1% /mnt/lustre2[OST:1] filesystem_summary: 524144 629 523515 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 30676 3576344 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 8256 3598764 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38932 7175108 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 350 999.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1000.0K 279 999.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 256.0K 402 255.6K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 256.0K 371 255.6K 1% /mnt/lustre2[OST:1] filesystem_summary: 511.9K 629 511.2K 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 350 1023650 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1024000 279 1023721 1% /mnt/lustre[MDT:1] filesystem_summary: 2048000 629 2047371 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 350 1023650 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1024000 279 1023721 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 262144 402 261742 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 262144 371 261773 1% /mnt/lustre2[OST:1] filesystem_summary: 524144 629 523515 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 30676 3576344 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 8256 3598764 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38932 7175108 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2056 1285632 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 30676 3576344 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 8256 3598764 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 38932 7175108 1% /mnt/lustre2 PASS 24a (1s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 18:00:57 (1748469657) PASS 24b (2s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 18:00:59 (1748469659) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (1s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 18:01:00 (1748469660) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] PASS 25b (2s) == sanityn test 26a: allow mtime to get older ============ 18:01:02 (1748469662) utime: good mknod times 1748469661 <= 1748469662 <= 1748469662 for /mnt/lustre/f26a utime: good mknod times 1748469661 <= 1748469662 <= 1748469662 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (2s) == sanityn test 26b: sync mtime between ost and mds ====== 18:01:04 (1748469664) PASS 26b (4s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 18:01:08 (1748469668) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7921 File: '/mnt/lustre/f26c.sanityn' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279612 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-05-28 18:01:08.000000000 -0400 Birth: - File: '/mnt/lustre2/f26c.sanityn' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279612 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-05-28 18:01:08.000000000 -0400 Birth: - PASS 26c (2s) SKIP: sanityn test_27 skipping excluded test 27 SKIP: sanityn test_28 skipping ALWAYS excluded test 28 == sanityn test 30: recreate file race =================== 18:01:11 (1748469671) striped dir -i0 -c2 -H crush /mnt/lustre/d30.sanityn 0 PASS 30 (4s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 18:01:15 (1748469675) striped dir -i1 -c2 -H crush /mnt/lustre/d31a.sanityn fail_loc=0x314 PASS 31a (2s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 18:01:17 (1748469677) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg357-server mds-ost sync done. Waiting for MDT destroys to complete striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d31b.sanityn fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.157@tcp:/lustre 7666232 38932 7174112 1% /mnt/lustre2 PASS 31b (10s) == sanityn test 31r: open-rename(replace) race =========== 18:01:27 (1748469687) -rw-r--r-- 1 root root 0 May 28 18:01 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 mv: failed to access '/mnt/lustre2/f31r.sanityn.target': Input/output error PASS 31r (4s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 18:01:31 (1748469691) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1213: 18919 Terminated tail -f $tfile PASS 31s (2s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 18:01:33 (1748469693) PASS 31t (3s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 18:01:36 (1748469696) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (1s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 18:01:37 (1748469697) Stopping /mnt/lustre-mds1 (opts:) on oleg357-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 create file on 2nd stripe under striped directory /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre/d33c.sanityn/striped touch /mnt/lustre2/d33c.sanityn/striped/subfile SoLC count 2 Below operations shouldn't trigger Solc: create file on 2nd stripe after setattr chmod 777 /mnt/lustre/d33c.sanityn/striped; touch /mnt/lustre2/d33c.sanityn/striped/subfile create remote dir and local dir /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote create local dir after remote dir creation transaction commit /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote.3; do_facet mds2 /home/green/git/lustre-release/lustre/utils/lctl set_param -n osd*.*MDT0001.force_sync 1; mkdir /mnt/lustre/d33c.sanityn/local.3 Solc count 0 PASS 33c (15s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 18:01:52 (1748469712) remote directory create /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre2/d33d.sanityn/subdir CoS count 2 cross-MDT rename mkdir /mnt/lustre/d33d.sanityn; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn/d1; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33d.sanityn/d2; touch /mnt/lustre/d33d.sanityn/d1/src mv /mnt/lustre2/d33d.sanityn/d1/src /mnt/lustre2/d33d.sanityn/d2/tgt CoS count 3 migrate /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs migrate -m 1 /mnt/lustre2/d33d.sanityn CoS count 2 PASS 33d (11s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 18:02:03 (1748469723) plain dir creation mkdir /mnt/lustre2/d33e.sanityn/plain open file and write echo abc > /mnt/lustre2/d33e.sanityn/f33e.sanityn append write echo dfg >> /mnt/lustre2/d33e.sanityn/f33e.sanityn setattr touch /mnt/lustre2/d33e.sanityn/f33e.sanityn file unlink rm /mnt/lustre2/d33e.sanityn/f33e.sanityn plain dir unlink rmdir /mnt/lustre2/d33e.sanityn/plain striped directory creation /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre2/d33e.sanityn/striped set default LMV to create striped subdir /home/green/git/lustre-release/lustre/utils/lfs setdirstripe -D -c 2 /mnt/lustre/d33e.sanityn striped subdir creation createmany -d /mnt/lustre/d33e.sanityn/subdir 100 total: 100 mkdir in 0.26 seconds: 384.34 ops/second sub file creation and write createmany -o /mnt/lustre/d33e.sanityn/subfile 100; echo abc > /mnt/lustre/d33e.sanityn/subfile1 total: 100 open/close in 0.25 seconds: 403.49 ops/second sub file append write echo dfg >> /mnt/lustre2/d33e.sanityn/subfile2 subdir setatttr touch /mnt/lustre2/d33e.sanityn/subdir1 subdir unlink unlinkmany -d /mnt/lustre/d33e.sanityn/subdir 100 - unlinked 0 (time 1748469725 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second sub file unlink unlinkmany /mnt/lustre2/d33e.sanityn/subfile 100 - unlinked 0 (time 1748469726 ; total 0 ; last 0) total: 100 unlinks in 0 seconds: inf unlinks/second sub file creation follows striped dir chmod chmod 777 /mnt/lustre/d33e.sanityn/striped; touch /mnt/lustre/d33e.sanityn/striped/subfile striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn/striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn CoS count 0 PASS 33e (5s) == sanityn test 34: no lock timeout under IO ============= 18:02:08 (1748469728) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg357-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800abd01000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800abd01000.ost_server_uuid in IDLE state after 0 sec oleg357-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800abd01000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800abd01000.ost_server_uuid in FULL state after 0 sec oleg357-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800abd01000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800abd01000.ost_server_uuid in IDLE state after 0 sec oleg357-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800abd01000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800abd01000.ost_server_uuid in FULL state after 0 sec oleg357-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800abd01000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800abd01000.ost_server_uuid in IDLE state after 0 sec oleg357-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800abd01000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800abd01000.ost_server_uuid in FULL state after 0 sec PASS 34 (85s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 18:03:33 (1748469813) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d35.sanityn fail_loc=0x80000317 Race attempt 0 Wait for 29516 29658 for 60 sec... - open/close 2718 (time 1748469825.92 total 10.00 last 271.78) total: 4000 open/close in 14.72 seconds: 271.68 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.157@tcp:/lustre 7.4G 39M 6.9G 1% /mnt/lustre 192.168.203.157@tcp:/lustre 7.4G 39M 6.9G 1% /mnt/lustre2 PASS 35 (65s) == sanityn test 36: handle ESTALE/open-unlink correctly == 18:04:38 (1748469878) striped dir -i0 -c2 -H all_char /mnt/lustre/d36.sanityn Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg357-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.316392 s, 166 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7921 Waiting for MDT destroys to complete *** cycle(0) *** before(7174104) after_dd(7122904) after(7174104) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.317262 s, 165 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7921 Waiting for MDT destroys to complete *** cycle(1) *** before(7174104) after_dd(7122904) after(7174104) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.308333 s, 170 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7921 Waiting for MDT destroys to complete *** cycle(2) *** before(7174104) after_dd(7122904) after(7174104) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.259819 s, 202 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7921 Waiting for MDT destroys to complete *** cycle(3) *** before(7174104) after_dd(7122904) after(7174104) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.3033 s, 173 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7921 Waiting for MDT destroys to complete *** cycle(4) *** before(7174104) after_dd(7122904) after(7174104) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.328906 s, 159 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7921 Waiting for MDT destroys to complete *** cycle(5) *** before(7174104) after_dd(7122904) after(7174104) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.376586 s, 139 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7921 Waiting for MDT destroys to complete *** cycle(6) *** before(7174104) after_dd(7122904) after(7174104) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.324336 s, 162 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7921 Waiting for MDT destroys to complete *** cycle(7) *** before(7174104) after_dd(7122904) after(7174104) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.309951 s, 169 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7921 Waiting for MDT destroys to complete *** cycle(8) *** before(7174104) after_dd(7122904) after(7174104) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.346745 s, 151 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7921 Waiting for MDT destroys to complete *** cycle(9) *** before(7174104) after_dd(7122904) after(7174104) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.341723 s, 153 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7921 Waiting for MDT destroys to complete *** cycle(10) *** before(7174104) after_dd(7122904) after(7174104) PASS 36 (175s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 18:07:33 (1748470053) striped dir -i1 -c2 -H crush2 /mnt/lustre/d37.sanityn multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7921 - create 7438 (time 1748470064.25 total 10.00 last 743.74) total: 10000 create in 13.30 seconds: 751.73 ops/second PASS 37 (16s) == sanityn test 39a: file mtime does not change after rename ========================================================== 18:07:49 (1748470069) repeat after cancel_lru_locks PASS 39a (1s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 18:07:50 (1748470070) repeat after cancel_lru_locks PASS 39b (3s) == sanityn test 39c: check truncate mtime update ================================================================================ 18:07:53 (1748470073) repeat after cancel_lru_locks PASS 39c (3s) == sanityn test 39d: sync write should update mtime ====== 18:07:56 (1748470076) fail_loc=0x411 fail_loc=0 PASS 39d (2s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 18:07:58 (1748470078) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=7 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (8s) == sanityn test 40c: pdirops: link and others ======================================================================== 18:08:06 (1748470086) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (7s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 18:08:13 (1748470093) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=8 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (7s) == sanityn test 40e: pdirops: rename and others ======================================================================== 18:08:20 (1748470100) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (6s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 18:08:26 (1748470106) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=11 mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (6s) == sanityn test 41b: pdirops: create vs create ======================================================================== 18:08:32 (1748470112) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 open: File exists Conflict PASS 41b (5s) == sanityn test 41c: pdirops: create vs link ======================================================================== 18:08:37 (1748470117) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (6s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 18:08:43 (1748470123) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 Conflict PASS 41d (6s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 18:08:49 (1748470129) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 41e (5s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 18:08:54 (1748470134) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 Conflict PASS 41f (6s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 18:09:00 (1748470140) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 Conflict PASS 41g (5s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 18:09:05 (1748470145) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 41h (6s) == sanityn test 41i: reint_open: create vs create ======== 18:09:11 (1748470151) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (695s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 18:20:46 (1748470846) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (5s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 18:20:51 (1748470851) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 42b (4s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 18:20:55 (1748470855) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (5s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 18:21:00 (1748470860) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42d (5s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 18:21:05 (1748470865) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (4s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 18:21:09 (1748470869) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 42f (4s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 18:21:13 (1748470873) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 Conflict PASS 42g (5s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 18:21:18 (1748470878) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42h (4s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 18:21:22 (1748470882) PASS 43a (20s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 18:21:42 (1748470902) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=5 Conflict PASS 43b (5s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 18:21:47 (1748470907) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43c (4s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 18:21:51 (1748470911) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (5s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 18:21:56 (1748470916) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 Conflict PASS 43e (4s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 18:22:00 (1748470920) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (4s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 18:22:04 (1748470924) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 stat: cannot stat '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (4s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 18:22:08 (1748470928) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 Conflict PASS 43h (4s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 18:22:12 (1748470932) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 43i (5s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 18:22:17 (1748470937) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (34s) == sanityn test 43k: unlink vs create ==================== 18:22:51 (1748470971) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (377s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 18:29:08 (1748471348) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Wed May 28 18:29:11 EDT 2025 PASS 44a (5s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 18:29:13 (1748471353) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 open: File exists Conflict PASS 44b (4s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 18:29:17 (1748471357) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (4s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 18:29:21 (1748471361) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=10 Conflict PASS 44d (5s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 18:29:26 (1748471366) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 Conflict PASS 44e (4s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 18:29:30 (1748471370) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=8 Conflict PASS 44f (4s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 18:29:34 (1748471374) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44g (5s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 18:29:39 (1748471379) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=7 Conflict PASS 44h (4s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 18:29:43 (1748471383) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 lfs mkdir: dirstripe error on '/mnt/lustre2/f44i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f44i.sanityn': File exists Conflict PASS 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 18:29:47 (1748471387) PASS 45a (30s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 18:30:17 (1748471417) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=81 Conflict PASS 45b (4s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 18:30:21 (1748471421) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=7 Conflict PASS 45c (5s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 18:30:26 (1748471426) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (4s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 18:30:30 (1748471430) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=7 Conflict PASS 45e (4s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 18:30:34 (1748471434) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=9 mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (5s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 18:30:39 (1748471439) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 stat: cannot stat '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (4s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 18:30:43 (1748471443) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=7 Conflict PASS 45h (4s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 18:30:47 (1748471447) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 45i (4s) == sanityn test 45j: read vs rename ====================== 18:30:51 (1748471451) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (434s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 18:38:05 (1748471885) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (5s) == sanityn test 46b: pdirops: link vs create ======================================================================== 18:38:10 (1748471890) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 open: File exists Conflict PASS 46b (5s) == sanityn test 46c: pdirops: link vs link ======================================================================== 18:38:16 (1748471896) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (6s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 18:38:21 (1748471901) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 46d (5s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 18:38:26 (1748471906) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 46e (6s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 18:38:32 (1748471912) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 Conflict PASS 46f (5s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 18:38:37 (1748471917) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=8 Conflict PASS 46g (6s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 18:38:43 (1748471923) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 46h (4s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 18:38:47 (1748471927) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 lfs mkdir: dirstripe error on '/mnt/lustre2/f46i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f46i.sanityn': File exists Conflict PASS 46i (5s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 18:38:52 (1748471932) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 mkdir: cannot create directory '/mnt/lustre2/f47a.sanityn': File exists Conflict PASS 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 18:38:56 (1748471936) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 open: File exists Conflict PASS 47b (6s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 18:39:02 (1748471942) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 link: cannot create link '/mnt/lustre2/f47c.sanityn' to '/mnt/lustre2/f47c.sanityn-2': File exists Conflict PASS 47c (4s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 18:39:06 (1748471946) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 47d (4s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 18:39:10 (1748471950) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f47e.sanityn' with non-directory Conflict PASS 47e (5s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 18:39:15 (1748471955) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 Conflict PASS 47f (4s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 18:39:20 (1748471960) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 47g (4s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 18:39:24 (1748471964) 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.00508021 s, 2.0 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (8s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 18:39:32 (1748471972) 0+1 records in 0+1 records out 159 bytes (159 B) copied, 0.00872493 s, 18.2 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3516: kill: (3537) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 18:39:36 (1748471976) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00563216 s, 182 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0237493 s, 44.2 MB/s 3145728 PASS 51b (14s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 18:39:50 (1748471990) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00670293 s, 153 kB/s PASS 51c (6s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 18:39:56 (1748471996) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0152706 s, 68.7 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (3s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 18:40:00 (1748472000) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1336 0x538 0x280000401 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1336 0x538 0x280000401 read lease(1) has applied. PASS 51e (3s) == sanityn test 54: rename locking ======================= 18:40:04 (1748472004) llite.lustre-ffff8800abd01000.xattr_cache=1 llite.lustre-ffff88012b480000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: '/mnt/lustre/d1/d2' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762663 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-28 18:40:04.000000000 -0400 Modify: 2025-05-28 18:40:04.000000000 -0400 Change: 2025-05-28 18:40:04.000000000 -0400 Birth: - ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: '/mnt/lustre/d1/d2' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281290 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-28 18:40:09.000000000 -0400 Modify: 2025-05-28 18:40:10.000000000 -0400 Change: 2025-05-28 18:40:10.000000000 -0400 Birth: - ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: '/mnt/lustre/d1/d2' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762666 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-28 18:40:15.000000000 -0400 Modify: 2025-05-28 18:40:15.000000000 -0400 Change: 2025-05-28 18:40:15.000000000 -0400 Birth: - open(O_RDWR|O_CREAT): Is a directory File: '/mnt/lustre/d1' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281291 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-28 18:40:15.000000000 -0400 Modify: 2025-05-28 18:40:15.000000000 -0400 Change: 2025-05-28 18:40:15.000000000 -0400 Birth: - ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: '/mnt/lustre/d1/d2' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281293 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-28 18:40:20.000000000 -0400 Modify: 2025-05-28 18:40:21.000000000 -0400 Change: 2025-05-28 18:40:21.000000000 -0400 Birth: - File: '/mnt/lustre/d1' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762667 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-28 18:40:20.000000000 -0400 Modify: 2025-05-28 18:40:21.000000000 -0400 Change: 2025-05-28 18:40:21.000000000 -0400 Birth: - PASS 54 (23s) == sanityn test 55a: rename vs unlink target dir ========= 18:40:28 (1748472028) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (8s) == sanityn test 55b: rename vs unlink source dir ========= 18:40:35 (1748472035) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55b.sanityn/d1/d2' to '/mnt/lustre/d55b.sanityn/d3/d2': No such file or directory PASS 55b (7s) == sanityn test 55c: rename vs unlink orphan target dir == 18:40:43 (1748472043) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 18:40:55 (1748472055) fail_loc=0x155 mv: cannot move '/mnt/lustre/d55d.sanityn/f1' to '/mnt/lustre/d55d.sanityn/d55d.sanityn': Is a directory PASS 55d (9s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 18:41:04 (1748472064) fail_loc=0x155 We survived after AB/BA race test fail_loc=0x0 PASS 55e (13s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 18:41:17 (1748472077) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (13s) == sanityn test 55g: rename: race with trylock =========== 18:41:30 (1748472090) mdt.lustre-MDT0000.enable_rename_trylock=1 mdt.lustre-MDT0001.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2': No such file or directory We survived race with trylock test fail_loc=0x0 PASS 55g (14s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 18:41:44 (1748472104) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1748472104 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 20.3464 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (5s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 18:41:49 (1748472109) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: read /mnt/lustre/f56b.sanityn@33554431995904+4096 no data: Success llverdev: read /mnt/lustre/f56b.sanityn@33554431995904+4096 no data: Success llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1748472110 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 5.81306 MB/s read offset: 32768000000kB -nan MB/s llverdev: data verified successfully PASS 56b (25s) == sanityn test 60: Verify data_version behaviour ======== 18:42:14 (1748472134) striped dir -i0 -c2 -H crush /mnt/lustre/d60.sanityn PASS 60 (2s) == sanityn test 70a: cd directory && rm directory ======== 18:42:16 (1748472136) PASS 70a (2s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 18:42:18 (1748472138) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 18:42:21 (1748472141) 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.0037546 s, 17.5 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00275528 s, 23.8 MB/s File: '/mnt/lustre2/f71a.sanityn' Size: 262144 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205289281333 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-28 18:42:21.000000000 -0400 Modify: 2025-05-28 18:42:21.000000000 -0400 Change: 2025-05-28 18:42:21.000000000 -0400 Birth: - 60163 extent 0 in offset 65536, length 65536 flags (0x80000000): NET extent 1 in offset 196608, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 2, sum of lengths 131072, expected sum 131072 60164 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00208013 s, 31.5 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00219601 s, 29.8 MB/s File: '/mnt/lustre/f71a.sanityn' Size: 262144 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205289281333 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-28 18:42:21.000000000 -0400 Modify: 2025-05-28 18:42:21.000000000 -0400 Change: 2025-05-28 18:42:21.000000000 -0400 Birth: - 60165 extent 0 in offset 65536, length 65536 flags (0x80000000): NET extent 1 in offset 196608, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 2, sum of lengths 131072, expected sum 131072 60164 PASS 71a (1s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 18:42:22 (1748472142) 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00611517 s, 10.7 MB/s extent 0 in offset 0, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 1, sum of lengths 65536, expected sum 65536 PASS 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 18:42:25 (1748472145) filefrag 1.46.2.wc5 (26-Mar-2022) check last flag for file with 291 extents 290: 296960.. 296963: 444416.. 444419: 4: 0000: last,net,eof check last flag for file with 292 extents 291: 297984.. 297987: 347136.. 347139: 4: 0001: last,net,eof check last flag for file with 293 extents 292: 299008.. 299011: 708608.. 708611: 4: 0000: last,net,eof PASS 71c (14s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 18:42:39 (1748472159) 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00909286 s, 7.2 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00239068 s, 27.4 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00247142 s, 26.5 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.0162988 s, 4.0 MB/s disk usage: 262144 /mnt/lustre/f71d.sanityn file size: 458752 /mnt/lustre/f71d.sanityn extent 0 in offset 0, length 65536 flags (0x80000000): NET extent 1 in offset 131072, length 65536 flags (0x80000000): NET extent 2 in offset 262144, length 65536 flags (0x80000000): NET extent 3 in offset 393216, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 4, sum of lengths 262144, expected sum 262144 PASS 71d (7s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 18:42:46 (1748472166) llite.lustre-ffff8800abd01000.xattr_cache=1 llite.lustre-ffff88012b480000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (3s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 18:42:49 (1748472169) llite.lustre-ffff8800abd01000.xattr_cache=1 llite.lustre-ffff88012b480000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (2s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 18:42:51 (1748472171) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00517056 s, 198 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00451711 s, 227 kB/s 20537: get lock2 20538: get lock1 20538: done 20538: get lock2 20537: done 20537: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 20537: cannot get lock1: Resource deadlock avoided fcntl cmd 7 failed: Resource deadlock avoided 20538: cannot get lock2: Resource deadlock avoided 20538: exit rc=0 20537: done 20537: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 18:42:57 (1748472177) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0136307 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0168452 s, 62.2 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4199: 21187 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 18:43:04 (1748472184) striped dir -i0 -c2 -H crush2 /mnt/lustre/d76.sanityn opening files: ................ closing files: ................ PASS 76 (44s) == sanityn test 77a: check FIFO NRS policy =============== 18:43:49 (1748472229) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0939691 s, 179 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00799821 s, 131 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00917835 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0224912 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0214735 s, 48.8 MB/s 1048576 bytes (1.0 MB) copied, 0.0197843 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0237869 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0193477 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0215279 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0332645 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0251749 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0269683 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0350157 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0339579 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0388561 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0344164 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011364 s, 92.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00973053 s, 108 MB/s 1048576 bytes (1.0 MB) copied, 0.00186061 s, 564 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00777913 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0140829 s, 74.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0152336 s, 68.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00161943 s, 647 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0136767 s, 76.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0129895 s, 80.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111624 s, 93.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00165967 s, 632 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0111862 s, 93.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133697 s, 78.4 MB/s , 0.00503097 s, 208 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00446692 s, 235 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0128712 s, 81.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0129255 s, 81.1 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg357-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77a.sanityn' PASS 77a (2s) == sanityn test 77b: check CRR-N NRS policy ============== 18:43:51 (1748472231) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.117314 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.018609 s, 56.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0204115 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0364659 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0492576 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0369578 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0335639 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0358392 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0476878 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0464589 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0456083 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0556077 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0413677 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0597633 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0591233 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0464182 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0571458 s, 18.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0103874 s, 101 MB/s , 0.0143153 s, 73.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111012 s, 94.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125295 s, 83.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00870762 s, 120 MB/s , 0.012365 s, 84.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111579 s, 94.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112664 s, 93.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011507 s, 91.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0166725 s, 62.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0131949 s, 79.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0135811 s, 77.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.014701 s, 71.3 MB/s , 0.0147535 s, 71.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0156355 s, 67.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0165049 s, 63.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=1 removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg357-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.103803 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0149817 s, 70.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0145072 s, 72.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0209598 s, 50.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0285655 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0276632 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0372916 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.025754 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0365323 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0421575 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0514583 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0542806 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0406401 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0506936 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0405979 s, 25.8 MB/s 1048576 bytes (1.0 MB) copied, 0.0414175 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0400814 s, 26.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0151906 s, 69.0 MB/s , 0.00964728 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00143807 s, 729 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0154049 s, 68.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0151295 s, 69.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0135095 s, 77.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0126687 s, 82.8 MB/s , 0.0111528 s, 94.0 MB/s 1048576 bytes (1.0 MB) copied, 0.00960326 s, 109 MB/s 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00311926 s, 336 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00867348 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00777872 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00744065 s, 141 MB/s , 0.0169912 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00620945 s, 169 MB/s 1048576 bytes (1.0 MB) copied, 0.00427648 s, 245 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=1 removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg357-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (3s) == sanityn test 77c: check ORR NRS policy ================ 18:43:54 (1748472234) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.102083 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00944381 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00757708 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0129123 s, 81.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0211779 s, 49.5 MB/s , 0.00992464 s, 106 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0219075 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0213806 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0154593 s, 67.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0264595 s, 39.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0265406 s, 39.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0236729 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0386123 s, 27.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0324453 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0460143 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0402493 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.045109 s, 23.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00704717 s, 149 MB/s , 0.00788584 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010228 s, 103 MB/s , 0.00760003 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00812916 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00824388 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00868544 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104658 s, 100 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107203 s, 97.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107552 s, 97.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0124177 s, 84.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120499 s, 87.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0129113 s, 81.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0123883 s, 84.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012256 s, 85.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120459 s, 87.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg357-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.148985 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107012 s, 98.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011152 s, 94.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110276 s, 95.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00698629 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0139448 s, 75.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0378171 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0322904 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0360449 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0332191 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0486523 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0412866 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0387395 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.043068 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0523346 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0507184 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0554029 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0096341 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00685889 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00820905 s, 128 MB/s 1048576 bytes (1.0 MB) copied, 0.0015557 s, 674 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00836189 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00835795 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110805 s, 94.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00925478 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00898115 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00941516 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110019 s, 95.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103825 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00965565 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113891 s, 92.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112866 s, 92.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01109 s, 94.6 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg357-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.10228 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103842 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0128817 s, 81.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.022634 s, 46.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.017424 s, 60.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0274961 s, 38.1 MB/s 1048576 bytes (1.0 MB) copied, 0.0227433 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0368022 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0279718 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0216756 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0078775 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0404028 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0340759 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0401739 s, 26.1 MB/s 1048576 bytes (1.0 MB) copied, 0.0310746 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0368736 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0324543 s, 32.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00847683 s, 124 MB/s 1048576 bytes (1.0 MB) copied, 0.0133612 s, 78.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00260108 s, 403 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00171432 s, 612 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00488034 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114158 s, 91.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00744491 s, 141 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00142691 s, 735 MB/s , 0.0198037 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104807 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00156788 s, 669 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00980947 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00588702 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00510467 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00791925 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00986647 s, 106 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg357-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (5s) == sanityn test 77d: check TRR nrs policy ================ 18:43:59 (1748472239) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.117965 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0147143 s, 71.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0131588 s, 79.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.017598 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0196831 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0182397 s, 57.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0338028 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0225026 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0216235 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00661458 s, 159 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.025582 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0413573 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.015077 s, 69.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0181204 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0403534 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0476605 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0217313 s, 48.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00628117 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00829575 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00134825 s, 778 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00270444 s, 388 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0079912 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105634 s, 99.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0012803 s, 819 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00980363 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01197 s, 87.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00954588 s, 110 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00988227 s, 106 MB/s , 0.0109357 s, 95.9 MB/s , 0.00993469 s, 106 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00205834 s, 509 MB/s , 0.00205814 s, 509 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00143332 s, 732 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg357-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.127443 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0166619 s, 62.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0161058 s, 65.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0267902 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0290794 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0193409 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0338609 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0373948 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0455927 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0494781 s, 21.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0355373 s, 29.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0456826 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0625793 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0519093 s, 20.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0547918 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0487486 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0730986 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00688643 s, 152 MB/s 1048576 bytes (1.0 MB) copied, 0.0070104 s, 150 MB/s , 0.00434565 s, 241 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111449 s, 94.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00909064 s, 115 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00678707 s, 154 MB/s , 0.00671983 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00703345 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00854336 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00925262 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100237 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101692 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110963 s, 94.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114074 s, 91.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125981 s, 83.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0134356 s, 78.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=1 removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg357-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.09951 s, 169 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00869805 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00972124 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0147304 s, 71.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.018348 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0271773 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0274858 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0272313 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0263144 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0266085 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0219498 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0272836 s, 38.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0449923 s, 23.3 MB/s , 0.0510032 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0411489 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0476598 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0527508 s, 19.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00157699 s, 665 MB/s , 0.00940209 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00434716 s, 241 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00891016 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0121182 s, 86.5 MB/s 1+0 records in 1+0 records out , 0.00788113 s, 133 MB/s 1048576 bytes (1.0 MB) copied, 0.00861678 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00662036 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00154387 s, 679 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00619682 s, 169 MB/s 1048576 bytes (1.0 MB) copied, 0.00768553 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00141075 s, 743 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00586876 s, 179 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00404011 s, 260 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00136481 s, 768 MB/s , 0.00313428 s, 335 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg357-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (4s) == sanityn test 77e: check TBF NID nrs policy ============ 18:44:03 (1748472243) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.57@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.111767 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116795 s, 89.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102985 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0314685 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0406232 s, 25.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0392734 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0413622 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0347594 s, 30.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0381653 s, 27.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0403775 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0420412 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0471208 s, 22.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.043478 s, 24.1 MB/s 1048576 bytes (1.0 MB) copied, 0.0487333 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0506143 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0481465 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0484237 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00690588 s, 152 MB/s 1048576 bytes (1.0 MB) copied, 0.00550051 s, 191 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109555 s, 95.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00733559 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00900294 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00753889 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00778691 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120207 s, 87.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00906576 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104882 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102305 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116001 s, 90.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0119151 s, 88.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118109 s, 88.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113397 s, 92.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0134804 s, 77.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=3 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg357-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.127248 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102447 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00922119 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0234307 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0184055 s, 57.0 MB/s , 0.0173996 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0183106 s, 57.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0333713 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0310324 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0217813 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0186625 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0342165 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0351927 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00850734 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0300524 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0407504 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0399871 s, 26.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00612572 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104187 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00574638 s, 182 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00192278 s, 545 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00641056 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00143205 s, 732 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0015039 s, 697 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0158892 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118256 s, 88.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00140127 s, 748 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117432 s, 89.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00130667 s, 802 MB/s , 0.0106402 s, 98.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00917063 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00393267 s, 267 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011019 s, 95.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg357-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.120298 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0171208 s, 61.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012262 s, 85.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0310729 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0198083 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0380122 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0239816 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0233891 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0469273 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.03151 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0382803 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0514534 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0448549 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0421283 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0358876 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0361042 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0333441 s, 31.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00927729 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106072 s, 98.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00765746 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118719 s, 88.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0134514 s, 78.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0086643 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0129769 s, 80.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106563 s, 98.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0114951 s, 91.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117284 s, 89.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125279 s, 83.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0123077 s, 85.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126555 s, 82.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133008 s, 78.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0136278 s, 76.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0146046 s, 71.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=1 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg357-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.168997 s, 99.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0145578 s, 72.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0219001 s, 47.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0403842 s, 26.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0212435 s, 49.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0382846 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0442862 s, 23.7 MB/s 1048576 bytes (1.0 MB) copied, 0.0424025 s, 24.7 MB/s 1048576 bytes (1.0 MB) copied, 0.0415425 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0459111 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0496364 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0443095 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0512874 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0587632 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.039134 s, 26.8 MB/s , 0.0470836 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0460929 s, 22.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00160016 s, 655 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0228144 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00145693 s, 720 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0189576 s, 55.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0227295 s, 46.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0198876 s, 52.7 MB/s , 0.0212675 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0135556 s, 77.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00989904 s, 106 MB/s , 0.00902187 s, 116 MB/s , 0.0183993 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00894731 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00891766 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00805172 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00302515 s, 347 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00737549 s, 142 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=2 removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg357-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' PASS 77e (7s) == sanityn test 77f: check TBF JobID nrs policy ========== 18:44:10 (1748472250) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg357-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.142171 s, 118 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0127929 s, 82.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124925 s, 83.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0136853 s, 76.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0348935 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0356677 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0418363 s, 25.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0453596 s, 23.1 MB/s 1048576 bytes (1.0 MB) copied, 0.0447648 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0475522 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0365431 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0483479 s, 21.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.05131 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.053398 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0493454 s, 21.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0437434 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0604057 s, 17.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=3 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00717052 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00926031 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00163512 s, 641 MB/s , 0.00970904 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00956233 s, 110 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00157677 s, 665 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00168761 s, 621 MB/s 1048576 bytes (1.0 MB) copied, 0.00271047 s, 387 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113563 s, 92.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.013347 s, 78.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118046 s, 88.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0123056 s, 85.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0133898 s, 78.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0128389 s, 81.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109148 s, 96.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125023 s, 83.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg357-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg357-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.141577 s, 119 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00984217 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112246 s, 93.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0164748 s, 63.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0206357 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0240299 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00809544 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100854 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0318602 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0155927 s, 67.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0335078 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.040725 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0394072 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0246186 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0269868 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0296459 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0421983 s, 24.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00179974 s, 583 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111145 s, 94.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00155381 s, 675 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00965706 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011748 s, 89.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102402 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0146622 s, 71.5 MB/s , 0.0135293 s, 77.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0123883 s, 84.6 MB/s , 0.00466295 s, 225 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126636 s, 82.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0130807 s, 80.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00205837 s, 509 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.013743 s, 76.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00319868 s, 328 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00517445 s, 203 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg357-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg357-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0981378 s, 171 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111978 s, 93.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00937511 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00883209 s, 119 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0355095 s, 29.5 MB/s , 0.0311608 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.027308 s, 38.4 MB/s , 0.0368744 s, 28.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0385796 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0396344 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0346977 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0306521 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.042092 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0413054 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0305078 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0381947 s, 27.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0434463 s, 24.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109007 s, 96.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104537 s, 100 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011617 s, 90.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0093026 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0154117 s, 68.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00951142 s, 110 MB/s 1048576 bytes (1.0 MB) copied, 0.0129271 s, 81.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0137526 s, 76.2 MB/s , 0.0131534 s, 79.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113771 s, 92.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0134523 s, 77.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0123357 s, 85.0 MB/s , 0.0123666 s, 84.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0144091 s, 72.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0147033 s, 71.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0160701 s, 65.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=3 removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg357-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg357-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.161837 s, 104 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0234082 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.024744 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0307428 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0255983 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0406125 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0398001 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0327815 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0435479 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0522711 s, 20.1 MB/s , 0.0373274 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0439548 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0421821 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0313793 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0384396 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0568573 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0570129 s, 18.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=3 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0110032 s, 95.3 MB/s , 0.010969 s, 95.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0100833 s, 104 MB/s 1048576 bytes (1.0 MB) copied, 0.00197318 s, 531 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120655 s, 86.9 MB/s , 0.00965198 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132951 s, 78.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104142 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00948776 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100662 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101707 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109469 s, 95.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109994 s, 95.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122234 s, 85.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125225 s, 83.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012493 s, 83.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg357-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' PASS 77f (8s) == sanityn test 77g: Change TBF type directly ============ 18:44:18 (1748472258) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (4s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 18:44:22 (1748472262) oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg357-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Unknown error 524 oleg357-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Unknown error 524 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 12 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Unknown error 524 oleg357-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Unknown error 524 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 12 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Unknown error 524 oleg357-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Unknown error 524 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 12 PASS 77h (4s) == sanityn test 77i: Change rank of TBF rule ============= 18:44:26 (1748472266) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (8s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 18:44:35 (1748472275) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.156239 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0146391 s, 71.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0231167 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0252339 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0317688 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0408 s, 25.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0398365 s, 26.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0502974 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.042474 s, 24.7 MB/s 1048576 bytes (1.0 MB) copied, 0.0425207 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0461523 s, 22.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0530839 s, 19.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0438983 s, 23.9 MB/s 1048576 bytes (1.0 MB) copied, 0.05604 s, 18.7 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0470288 s, 22.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0421181 s, 24.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0560922 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.173683 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.175152 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.173746 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.174706 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.177477 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.176967 s, 5.9 MB/s , 0.177949 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.177852 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.180641 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.179805 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.181045 s, 5.8 MB/s , 0.182041 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.181645 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.182784 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.184979 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.183941 s, 5.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=1 removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg357-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.91829 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.4541 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (41s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 18:45:15 (1748472315) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg357-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg357-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg357-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg357-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg357-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.138427 s, 121 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0150514 s, 69.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108551 s, 96.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112955 s, 92.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0243895 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124735 s, 84.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0302677 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0343473 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0415411 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118016 s, 88.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0129452 s, 81.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125513 s, 83.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0187518 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0583058 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.100802 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0640572 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.108976 s, 9.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106359 s, 98.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0179905 s, 58.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0164247 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0195558 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0196165 s, 53.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0154867 s, 67.7 MB/s , 0.0178689 s, 58.7 MB/s 1048576 bytes (1.0 MB) copied, 0.0191808 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0161597 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0161379 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0161336 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0163578 s, 64.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00206332 s, 508 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0161791 s, 64.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0175607 s, 59.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0188641 s, 55.6 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg357-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.6648 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9496 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg357-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg357-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg357-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg357-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg357-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.136455 s, 123 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0117481 s, 89.3 MB/s , 0.0143486 s, 73.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0183427 s, 57.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0274769 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0289703 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120894 s, 86.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0225615 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106447 s, 98.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0615577 s, 17.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0439187 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0514789 s, 20.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0669575 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0383791 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0546185 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0441317 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0538148 s, 19.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00913032 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00914526 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010643 s, 98.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00971577 s, 108 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0106031 s, 98.9 MB/s , 0.0116572 s, 90.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0105376 s, 99.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107198 s, 97.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012557 s, 83.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111061 s, 94.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132904 s, 78.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0128542 s, 81.6 MB/s 1048576 bytes (1.0 MB) copied, 0.0123321 s, 85.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124353 s, 84.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121545 s, 86.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0147523 s, 71.1 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg357-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.832 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9453 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (110s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 18:47:05 (1748472425) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg357-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg357-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg357-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg357-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg357-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.176391 s, 95.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0181761 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0159154 s, 65.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0227623 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0228739 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0242865 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111706 s, 93.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0464627 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0188511 s, 55.6 MB/s , 0.0203281 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104222 s, 101 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0194034 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.015681 s, 66.9 MB/s 1048576 bytes (1.0 MB) copied, 0.0214626 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0160631 s, 65.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0127453 s, 82.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0685819 s, 15.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0174224 s, 60.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0168484 s, 62.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0175244 s, 59.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0160967 s, 65.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0197733 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0216486 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0203137 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.015675 s, 66.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.017786 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0178707 s, 58.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0175453 s, 59.8 MB/s 1048576 bytes (1.0 MB) copied, 0.0185635 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0182387 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0186374 s, 56.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0163571 s, 64.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0200582 s, 52.3 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg357-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.6635 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9526 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg357-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg357-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg357-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg357-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg357-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.122524 s, 137 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.012084 s, 86.8 MB/s 1048576 bytes (1.0 MB) copied, 0.0108642 s, 96.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0154843 s, 67.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0129564 s, 80.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0181992 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0208106 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00825232 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0270825 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0303413 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00895288 s, 117 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0224117 s, 46.8 MB/s 1048576 bytes (1.0 MB) copied, 0.0525289 s, 20.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0219068 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0529657 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.026212 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0290457 s, 36.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0112332 s, 93.3 MB/s 1048576 bytes (1.0 MB) copied, 0.00768641 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00824161 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00995226 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116084 s, 90.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0150401 s, 69.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0136295 s, 76.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01241 s, 84.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.010865 s, 96.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0109652 s, 95.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107093 s, 97.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121525 s, 86.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125863 s, 83.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106421 s, 98.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112818 s, 92.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0147179 s, 71.2 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg357-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.6625 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.942 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (111s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 18:48:56 (1748472536) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg357-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.127224 s, 132 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102371 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01171 s, 89.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0281369 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0217736 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0321565 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0215719 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0323228 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0286228 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.039544 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.040364 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0448865 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0407391 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0431875 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0439745 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.042224 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0521691 s, 20.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00798976 s, 131 MB/s , 0.00710803 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00967166 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00932378 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00841311 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00887113 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00292195 s, 359 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00239891 s, 437 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00936521 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00819617 s, 128 MB/s 1048576 bytes (1.0 MB) copied, 0.00954381 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108759 s, 96.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00921382 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101015 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00983879 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010387 s, 101 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg357-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.89074 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.73619 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.57@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.57@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.109324 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010891 s, 96.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0127028 s, 82.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0294993 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0222309 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0337377 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0285329 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0331052 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0363914 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0493544 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0494505 s, 21.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0383276 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0473965 s, 22.1 MB/s 1048576 bytes (1.0 MB) copied, 0.0454567 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0421097 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0466931 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0568508 s, 18.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0820697 s, 12.8 MB/s 1048576 bytes (1.0 MB) copied, 0.0850158 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0872175 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0835417 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0868525 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0851372 s, 12.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0844692 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0871572 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0860207 s, 12.2 MB/s , 0.087892 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0870032 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0864929 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0874763 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0877927 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0878305 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.089979 s, 11.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=1 removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg357-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.89655 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.74692 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.57@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg357-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0930457 s, 180 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0098714 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0158505 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.016312 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0298405 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0260643 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0273727 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0345174 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0280416 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0393258 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0385182 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0267823 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0344257 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.041698 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0344258 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0373039 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0381839 s, 27.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00142014 s, 738 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0085677 s, 122 MB/s 1048576 bytes (1.0 MB) copied, 0.00675589 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00766429 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00789087 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00119164 s, 880 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00166822 s, 629 MB/s 1048576 bytes (1.0 MB) copied, 0.0044988 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00301185 s, 348 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00911508 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00112147 s, 935 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00979415 s, 107 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0107195 s, 97.8 MB/s , 0.010728 s, 97.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103136 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00866441 s, 121 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg357-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.84907 s, 21.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.88219 s, 21.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg357-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.13621 s, 123 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100423 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125863 s, 83.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0180337 s, 58.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0309871 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0281857 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0369593 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0318537 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.030025 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0390631 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0415636 s, 25.2 MB/s 1048576 bytes (1.0 MB) copied, 0.03933 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0372761 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0405308 s, 25.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0340316 s, 30.8 MB/s , 0.0360829 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0380491 s, 27.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0019866 s, 528 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00659499 s, 159 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100448 s, 104 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00721998 s, 145 MB/s , 0.00291195 s, 360 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00274191 s, 382 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00814241 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00987802 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103424 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104892 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108949 s, 96.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124547 s, 84.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012678 s, 82.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012 s, 87.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126564 s, 82.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124432 s, 84.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg357-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.73827 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.72644 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.87911 s, 21.5 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.73499 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg357-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.121954 s, 138 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00977939 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00982683 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00688453 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0159802 s, 65.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00814461 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00619636 s, 169 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0242333 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00981551 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0233934 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0322956 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0332512 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0330684 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0432758 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0518379 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0491592 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0359377 s, 29.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00726759 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0077592 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00298198 s, 352 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00935793 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00908845 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00665228 s, 158 MB/s , 0.0111705 s, 93.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00793285 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103437 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00884982 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00949541 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00920989 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00873216 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011108 s, 94.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101959 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0140949 s, 74.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg357-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.6521 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.4434 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg357-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0934447 s, 180 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00977598 s, 107 MB/s , 0.00940536 s, 111 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0253049 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0320581 s, 32.7 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0360577 s, 29.1 MB/s 1048576 bytes (1.0 MB) copied, 0.0290995 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.027078 s, 38.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0404055 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.046811 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0397798 s, 26.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0371632 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0450556 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0460511 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0532955 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0510636 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0473149 s, 22.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00998225 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00905154 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00955144 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00830036 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122899 s, 85.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108015 s, 97.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0148557 s, 70.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115736 s, 90.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0103133 s, 102 MB/s 1+0 records in 1+0 records out , 0.0116749 s, 89.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115416 s, 90.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118632 s, 88.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0121217 s, 86.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0136629 s, 76.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0137987 s, 76.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0143724 s, 73.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg357-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.90409 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.75082 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg357-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0926247 s, 181 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00695113 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00945482 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126625 s, 82.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0246092 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0259989 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0219099 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0316299 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0337696 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0364945 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0327168 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0286514 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0375762 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0275742 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.03215 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0337934 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0376879 s, 27.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100504 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0078504 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00863984 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00832739 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100917 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00887297 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00826184 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00841843 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100091 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107042 s, 98.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0098798 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103282 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00856359 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00804308 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00940874 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107141 s, 97.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg357-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.74569 s, 10.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.7331 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.73978 s, 10.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.72996 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (264s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 18:53:21 (1748472801) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (3s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 18:53:23 (1748472803) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg357-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 3.11631 s, 336 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg357-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 40.009 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (50s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 18:54:13 (1748472853) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.120645 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00831466 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0138196 s, 75.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0247813 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0146312 s, 71.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0162897 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0210051 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0326949 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0338909 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0164222 s, 63.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0379454 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0362762 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0396113 s, 26.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0418757 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00946442 s, 111 MB/s 1048576 bytes (1.0 MB) copied, 0.0370224 s, 28.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0450506 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00871901 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00144075 s, 728 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00693721 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.004356 s, 241 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00857866 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00342676 s, 306 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00138021 s, 760 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114837 s, 91.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0221781 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0172723 s, 60.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00552039 s, 190 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0147793 s, 70.9 MB/s , 0.00470354 s, 223 MB/s 1048576 bytes (1.0 MB) copied, 0.0122055 s, 85.9 MB/s 1048576 bytes (1.0 MB) copied, 0.00356604 s, 294 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110589 s, 94.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012b480000.lock_unused_count=1 removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg357-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.90574 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.97278 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0977703 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00843519 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108159 s, 96.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0267271 s, 39.2 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0252725 s, 41.5 MB/s , 0.0233755 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0277212 s, 37.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0267506 s, 39.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0317755 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0246483 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012805 s, 81.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108162 s, 96.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0360119 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0335297 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0359929 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0252488 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0498806 s, 21.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012b480000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0738721 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0738385 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0724731 s, 14.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0770396 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0741594 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.074965 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0749693 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0763483 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0796293 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0758018 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0777939 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0799935 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0783426 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0786016 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0804685 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0827607 s, 12.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg357-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg357-client.virtnet' removed directory: '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.91085 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.97244 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (43s) == sanityn test 77o: Changing rank should not panic ====== 18:54:56 (1748472896) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 18:55:00 (1748472900) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (32s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 18:55:32 (1748472932) mds.MDS.mdt.nrs_policies=tbf oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg357-client: oleg357-server: ssh exited with exit code 36 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg357-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (13s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 18:55:45 (1748472945) striped dir -i0 -c1 -H all_char /mnt/lustre/d77r.sanityn mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 20360 kill md_thread xargs PASS 77r (38s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 18:56:23 (1748472983) oleg357-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg357-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg357-client: oleg357-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 18:56:26 (1748472986) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d79.sanityn File: '/mnt/lustre/d79.sanityn' Size: 8192 Blocks: 16 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798763876 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-28 18:56:27.000000000 -0400 Modify: 2025-05-28 18:56:27.000000000 -0400 Change: 2025-05-28 18:56:27.000000000 -0400 Birth: - fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 18:56:39 (1748472999) total: 10 open/close in 0.03 seconds: 392.74 ops/second multiop /mnt/lustre2/d80a.sanityn/f80a.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7921 Finish migration, then checking.. PASS 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 18:56:42 (1748473002) start migration thread 31934 accessing the migrating directory for 1 minute... concurrent access 1986 failures, 2550 successes /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 31934 Killed ( while true; do mdt_idx=$((RANDOM % MDSCOUNT)); $LFS migrate -m $mdt_idx $migdir1 &>/dev/null; done ) (wd: ~) PASS 80b (62s) == sanityn test 81a: rename and stat under striped directory ========================================================== 18:57:44 (1748473064) stat: cannot stat 'd0/0': No such file or directory File: 'd0/0' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289284603 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-28 18:57:44.000000000 -0400 Modify: 2025-05-28 18:57:44.000000000 -0400 Change: 2025-05-28 18:57:45.000000000 -0400 Birth: - PASS 81a (2s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 18:57:46 (1748473066) total: 2500 open/close in 8.25 seconds: 303.02 ops/second /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 6179 Killed ( while true; do touch $DIR1/$tdir; done ) (wd: ~) PASS 81b (52s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 18:58:38 (1748473118) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 18:58:39 (1748473119) - open/close 2452 (time 1748473131.52 total 10.00 last 245.19) total: 2500 open/close in 10.20 seconds: 245.16 ops/second total: 2500 open/close in 7.52 seconds: 332.62 ops/second snapshot_time 1748473166.958674013 secs.nsecs start_time 1748473121.408074662 secs.nsecs elapsed_time 45.550599351 secs.nsecs open 2505 samples [usecs] 55 1602 899471 343212041 close 2504 samples [usecs] 41 869 259733 28866195 mknod 2501 samples [usecs] 130 1528 757028 245388548 unlink 2500 samples [usecs] 159 12781 1205798 2648352360 mkdir 2 samples [usecs] 525 2742 3267 7794189 rmdir 2 samples [usecs] 2349 3963 6312 21223170 rename 2500 samples [usecs] 153 8439 1733228 2072166146 getattr 3764 samples [usecs] 0 581 56213 1502333 statfs 10 samples [usecs] 8 17 124 1634 parallel_rename_file 2500 samples [usecs] 153 8439 1733228 2072166146 crossdir_rename 2500 samples [usecs] 153 8439 1733228 2072166146 snapshot_time 1748473166.958872476 secs.nsecs start_time 1748473121.408244425 secs.nsecs elapsed_time 45.550628051 secs.nsecs open 2504 samples [usecs] 37 1230 674237 208447897 close 2504 samples [usecs] 38 1080 193927 18167241 mknod 2500 samples [usecs] 114 1191 566170 148008148 unlink 2500 samples [usecs] 153 26143 1186649 2872229067 mkdir 2 samples [usecs] 505 6648 7153 44450929 rmdir 2 samples [usecs] 2661 7567 10228 64340410 rename 2500 samples [usecs] 151 3332 3200304 7105679606 getattr 3757 samples [usecs] 5 244 83308 2477502 statfs 10 samples [usecs] 6 18 130 1796 parallel_rename_file 2500 samples [usecs] 151 3332 3200304 7105679606 crossdir_rename 2500 samples [usecs] 151 3332 3200304 7105679606 PASS 81d (49s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 18:59:28 (1748473168) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7921 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7921 PASS 82 (2s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 18:59:30 (1748473170) start pid 21105 to create/unlink striped directory start pid 21106 to stat striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5664: 21105 Terminated ( cd $DIR1; while true; do $LFS mkdir -i1 -c2 $tdir; rmdir $tdir; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5664: 21106 Terminated ( cd $DIR2; while true; do stat $tdir > /dev/null 2>&1; done ) PASS 83 (123s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 19:01:33 (1748473293) fail_loc=0x8000060b PASS 84 (7s) == sanityn test 85: Lustre API root cache race =========== 19:01:40 (1748473300) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1748473301 Finishing test test1 at 1748473302 PASS 85 (4s) == sanityn test 90: open/create and unlink striped directory ========================================================== 19:01:44 (1748473304) start pid 1188 to open/create under striped directory start pid 1191 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5731: 1188 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; touch $tdir/f{0..3} > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5731: 1191 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 90 (182s) == sanityn test 91: chmod and unlink striped directory === 19:04:46 (1748473486) start pid 11701 to chmod striped directory start pid 11702 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5768: 11701 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; chmod go+w $tdir > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5768: 11702 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 91 (182s) == sanityn test 92: create remote directory under orphan directory ========================================================== 19:07:48 (1748473668) fail_loc=0x1408 mkdir: cannot create directory '/mnt/lustre2/d92.sanityn/dir': No such file or directory lfs setdirstripe: cannot resolve path '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory (2) lfs setdirstripe: '/mnt/lustre2/d92.sanityn/remote_dir' is not on a Lustre filesystem: No such file or directory (2) lfs setdirstripe: cannot create dir '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory fail_loc=0 PASS 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 19:07:51 (1748473671) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6604 0x19cc 0x280000401 1 6593 0x19c1 0x2c0000401 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6594 0x19c2 0x2c0000401 0 6605 0x19cd 0x280000401 rc1=0 and rc2=0 PASS 93 (9s) == sanityn test 94: signal vs CP callback race =========== 19:08:00 (1748473680) osc.lustre-OST0000-osc-ffff8800abd01000.idle_timeout=0 osc.lustre-OST0000-osc-ffff88012b480000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8800abd01000.idle_timeout=0 osc.lustre-OST0001-osc-ffff88012b480000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0132471 s, 618 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 3.00253 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5855: 27201 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00869421 s, 471 kB/s osc.lustre-OST0000-osc-ffff8800abd01000.idle_timeout=debug osc.lustre-OST0000-osc-ffff88012b480000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800abd01000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012b480000.idle_timeout=debug PASS 94 (11s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 19:08:11 (1748473691) llite.lustre-ffff8800abd01000.fast_read=0 llite.lustre-ffff88012b480000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0239923 s, 87.4 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8800abd01000.fast_read=1 llite.lustre-ffff88012b480000.fast_read=1 PASS 95a (19s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 19:08:30 (1748473710) llite.lustre-ffff8800abd01000.fast_read=0 llite.lustre-ffff88012b480000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB) copied, 0.00279721 s, 4.4 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8800abd01000.fast_read=1 llite.lustre-ffff88012b480000.fast_read=1 PASS 95b (9s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 19:08:39 (1748473719) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 19:08:40 (1748473720) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00270388 s, 1.5 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (3s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 19:08:43 (1748473723) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0321092 s, 65.3 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 19:08:45 (1748473725) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0319385 s, 65.7 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 19:08:48 (1748473728) 1+0 records in 1+0 records out 12000 bytes (12 kB) copied, 0.00278894 s, 4.3 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 19:08:50 (1748473730) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.000834667 s, 4.9 MB/s PASS 101a (2s) == sanityn test 101b: Discard DoM data on rename ========= 19:08:52 (1748473732) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.000873034 s, 4.7 MB/s PASS 101b (3s) == sanityn test 101c: Discard DoM data on close-unlink === 19:08:55 (1748473735) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.000891337 s, 4.6 MB/s PASS 101c (3s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 19:08:58 (1748473738) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 00 02 00 00 00 80 31 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000402:0x3180:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.remote fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 40 02 00 00 00 34 3e 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000402:0x3e34:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.remote/subdir fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 40 02 00 00 00 35 3e 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000402:0x3e35:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.stripe fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 00 02 00 00 00 81 31 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000402:0x3181:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.stripe/subdir fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 00 02 00 00 00 82 31 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000402:0x3182:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! PASS 102 (3s) == sanityn test 103: Test size correctness with lockahead ========================================================== 19:09:01 (1748473741) striped dir -i1 -c2 -H crush /mnt/lustre/d103.sanityn fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1748473742 lockahead_test: lockahead_test.c:1191: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1748473745 Finishing test test23 at 1748473745 PASS 103 (8s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 19:09:09 (1748473749) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00457979 s, 224 kB/s oleg357-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg357-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps STAT a|m|ctime 1748473750 1748473750 1748473750 MDT a|m|ctime 1748473750 1748473750 1748473750 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00370274 s, 277 kB/s oleg357-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg357-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps STAT a|m|ctime 1748473750 1748473753 1748473753 MDT a|m|ctime 1748473750 1748473753 1748473753 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0110692 s, 92.5 kB/s oleg357-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg357-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps STAT a|m|ctime 1748473750 1748473757 1748473757 MDT a|m|ctime 1748473750 1748473757 1748473757 oleg357-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg357-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps oleg357-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg357-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps STAT a|m|ctime 1748473750 1748473757 1748473757 MDT a|m|ctime 1748473750 1748473757 1748473757 oleg357-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg357-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps STAT a|m|ctime 1748473750 1748473757 1748473766 MDT a|m|ctime 1748473750 1748473757 1748473766 PASS 104 (20s) == sanityn test 105: Glimpse and lock cancel race ======== 19:09:29 (1748473769) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d105.sanityn fail_loc=0x416 PASS 105 (28s) == sanityn test 106a: Verify the btime via statx() ======= 19:09:57 (1748473797) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 19:09:58 (1748473798) SKIP: sanityn test_106b statx() only test SKIP 106b (1s) == sanityn test 106c: Verify statx attributes mask ======= 19:09:59 (1748473799) SKIP: sanityn test_106c statx() only test SKIP 106c (1s) == sanityn test 107a: Basic grouplock conflict =========== 19:10:00 (1748473800) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.079178 s, 132 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6600 0x19c8 0x2c0000401 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7921 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7921 First grouplock blocks second one PASS 107a (5s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 19:10:05 (1748473805) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0991048 s, 106 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6611 0x19d3 0x280000401 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7921 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (8s) == sanityn test 108a: lseek: parallel updates ============ 19:10:13 (1748473813) 1+0 records in 1+0 records out 10000 bytes (10 kB) copied, 0.00272344 s, 3.7 MB/s fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB) copied, 4.15598 s, 2.0 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 19:10:20 (1748473820) 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 3 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 11 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 13 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 14 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 15 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 16 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 18 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 19 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 21 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg357-server: libkmod: kmod_module_get_size: failed to read coresize from /sys/module/intel_rapl oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 22 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 24 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: libkmod: kmod_module_get_size: failed to read coresize from /sys/module/acpi_cpufreq Iteration 25 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 27 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 28 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 32 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 37 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 38 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 42 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 43 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg357-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 46 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 192.168.203.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.157@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg357-client.virtnet /mnt/lustre (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre2 PASS 109 (481s) == sanityn test 110: do not grant another lock on resend ========================================================== 19:18:21 (1748474301) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: '/mnt/lustre/f110.sanityn' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934275 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-28 19:18:21.000000000 -0400 Modify: 2025-05-28 19:18:21.000000000 -0400 Change: 2025-05-28 19:18:21.000000000 -0400 Birth: - fail_loc=0 fail_val=0 Connected clients: oleg357-client.virtnet oleg357-client.virtnet PASS 110 (65s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 19:19:26 (1748474366) fail_loc=0x8000018a /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6644: 31588 Terminated $MULTIOP $DIR2/$tdir/mdt0dir/foodir/file2 Ow4096_c PASS 111 (6s) == sanityn test 112: update max-inherit in default LMV === 19:19:32 (1748474372) lmv.lustre-clilmv-ffff8800b3a27000.qos_threshold_rr=90 lmv.lustre-clilmv-ffff8800b5990800.qos_threshold_rr=90 PASS 112 (4s) == sanityn test 113: check servers of specified fs ======= 19:19:36 (1748474376) lustre-OST0000-osc-ffff8800b5990800 active. lustre-OST0001-osc-ffff8800b5990800 active. lustre-OST0000-osc-ffff8800b5990800 active. lustre-OST0001-osc-ffff8800b5990800 active. MGC192.168.203.157@tcp active. PASS 113 (3s) == sanityn test 114: implicit default LMV inherit ======== 19:19:39 (1748474379) /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 PASS 114 (8s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 19:19:47 (1748474387) total: 3000 create in 6.42 seconds: 467.62 ops/second oleg357-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg357-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps oleg357-server: htree_dump_int_block: Ext2 directory block not found while calculating checksum for htree root oleg357-server: USE: f1863 f70 fail_loc=0x80002401 fail_val=5 mkdir: cannot create directory '/mnt/lustre/d115.sanityn/f70': File exists PASS 115 (15s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 19:20:02 (1748474402) llite.lustre-ffff8800b3a27000.intent_mkdir=1 llite.lustre-ffff8800b5990800.intent_mkdir=1 MD layout /mnt/lustre/d116.sanityn: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000bd0:0x78c:0x0] 1 [0x240000404:0x70e:0x0] mkdir /mnt/lustre/d116.sanityn/tdir0 setdirstripe -D -i1 /mnt/lustre2/d116.sanityn/tdir0 mkdir /mnt/lustre/d116.sanityn/tdir0/tdir11 lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none mdtidx FID[seq:oid:ver] lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none mdtidx FID[seq:oid:ver] PASS 116 (3s) == sanityn test 200: service remains healthy while able to process request ========================================================== 19:20:05 (1748474405) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=18 striped dir -i0 -c2 -H crush2 /mnt/lustre/d200.sanityn fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.0175 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.0052 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.01 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.01 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.1084 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.1068 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.1065 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.1056 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.105 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied4096 bytes (4.1 kB) copied, 40.005 s, 0.1 kB/s , 39.9904 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.0889 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.08 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.1054 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.1755 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.1661 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.168 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.1919 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.1168 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.0918 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.1628 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.1861 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.1705 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.2455 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.2441 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.2428 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.2456 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 80.1542 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 80.1733 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 80.261 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 80.2532 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 80.2469 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 80.3144 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 80.3387 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 80.3567 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 80.3413 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg357-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b3a27000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b3a27000.ost_server_uuid in IDLE state after 0 sec PASS 200 (87s) cleanup: ====================================================== == sanityn test complete, duration 5491 sec ============== 19:21:33 (1748474493) === sanityn: start cleanup 19:21:33 (1748474493) === Stopping clients: oleg357-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg357-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 19:23:19 (1748474599) ===