-----============= acceptance-small: sanityn ============----- Tue Apr 8 15:28:29 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg213-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanityn.*ex: No such file or directory excepting tests: 27 28 skipping tests SLOW=no: 33a === sanityn: start setup 15:28:36 (1744140516) === Starting client oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 Started clients oleg213-client.virtnet: 192.168.202.113@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) oleg213-client.virtnet: executing check_config_client /mnt/lustre oleg213-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg213-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800a9803000.idle_timeout=debug osc.lustre-OST0000-osc-ffff88012a6ad800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800a9803000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a6ad800.idle_timeout=debug disable quota as required oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanityn: finish setup 15:28:46 (1744140526) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0119559 s, 87.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7933] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 15:28:47 (1744140527) PASS 0 (2s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 15:28:50 (1744140530) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (1s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 15:28:51 (1744140531) -rw-r--r-- 1 root root 0 Apr 8 15:28 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (2s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 15:28:53 (1744140533) -rw-r--r-- 1 root root 0 Apr 8 15:28 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (2s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 15:28:55 (1744140535) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (1s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 15:28:56 (1744140536) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (2s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 15:28:58 (1744140538) total 0 -rwxrwxrwx 1 root root 0 Apr 8 15:28 f2a -rwxrwxrwx 1 root root 0 Apr 8 15:28 f2b total 0 -rwxrwxrwx 1 root root 0 Apr 8 15:28 f2a -rwxrwxrwx 1 root root 0 Apr 8 15:28 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.00368322 s, 139 kB/s PASS 2e (2s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 15:29:00 (1744140540) /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has type file OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has perms 0777 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by user #500 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by group #500 OK Can't lstat /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn: No such file or directory PASS 2f (2s) == sanityn test 2g: check blocks update on sync write ==== 15:29:02 (1744140542) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0665466 s, 31.5 MB/s /mnt/lustre/f2g.sanityn has 4096 blocks /mnt/lustre2/f2g.sanityn has 4096 blocks PASS 2g (2s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 15:29:04 (1744140544) PASS 3 (2s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 15:29:06 (1744140546) PASS 4 (2s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 15:29:08 (1744140548) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (2s) == sanityn test 6: remove of open file on other node ============================================================================ 15:29:10 (1744140550) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (2s) == sanityn test 7: remove of open directory on other node ======================================================================= 15:29:12 (1744140552) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (1s) == sanityn test 8: remove of open special file on other node ==================================================================== 15:29:13 (1744140553) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (2s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 15:29:15 (1744140555) PASS 9a (3s) == sanityn test 9b: append to striped sparse file ======== 15:29:18 (1744140558) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0175452 s, 59.8 MB/s 3+0 records in 3+0 records out 3 bytes (3 B) copied, 0.0196669 s, 0.2 kB/s Data read (expecting 'foo'): foo PASS 9b (2s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 15:29:20 (1744140560) 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00493121 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00338216 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00430126 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00455719 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00475891 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00536932 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00566738 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00900377 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00380295 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00352399 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00446218 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00428376 s, 0.2 kB/s PASS 10a (2s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 15:29:22 (1744140562) 1+0 records in 1+0 records out 3072 bytes (3.1 kB) copied, 0.00843247 s, 364 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0107513 s, 381 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB) copied, 0.000194064 s, 15.8 MB/s PASS 10b (2s) == sanityn test 11: execution of file opened for write should return error ============================================================== 15:29:24 (1744140564) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d11 multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7933 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 301: /mnt/lustre2/d11/f: Text file busy PASS 11 (2s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 15:29:26 (1744140566) warning: '-runtime' deprecated, use '-t runtime' instead - link 1309 (time 1744140576.95 total 10.00 last 130.88) - link 2797 (time 1744140586.96 total 20.01 last 148.72) - link 4494 (time 1744140596.96 total 30.01 last 169.67) - link 6187 (time 1744140606.96 total 40.01 last 169.24) - link 8046 (time 1744140616.97 total 50.01 last 185.85) - link 10000 (time 1744140626.64 total 59.69 last 202.02) - link 11835 (time 1744140636.64 total 69.69 last 183.48) - link 13793 (time 1744140646.64 total 79.69 last 195.79) - link 15738 (time 1744140656.64 total 89.69 last 194.46) - link 17908 (time 1744140666.65 total 99.69 last 216.94) total: 17991 link in 100.00 seconds: 179.90 ops/second using seed 845552050 running for 100 seconds - stat 10000 (time 1744140571 ; total 4 ; last 4) - stat 20000 (time 1744140574 ; total 7 ; last 3) - stat 30000 (time 1744140578 ; total 11 ; last 4) - stat 40000 (time 1744140581 ; total 14 ; last 3) - stat 50000 (time 1744140585 ; total 18 ; last 4) - stat 60000 (time 1744140588 ; total 21 ; last 3) - stat 70000 (time 1744140591 ; total 24 ; last 3) - stat 80000 (time 1744140594 ; total 27 ; last 3) - stat 90000 (time 1744140598 ; total 31 ; last 4) - stat 100000 (time 1744140601 ; total 34 ; last 3) - stat 110000 (time 1744140604 ; total 37 ; last 3) - stat 120000 (time 1744140607 ; total 40 ; last 3) - stat 130000 (time 1744140610 ; total 43 ; last 3) - stat 140000 (time 1744140614 ; total 47 ; last 4) - stat 150000 (time 1744140617 ; total 50 ; last 3) - stat 160000 (time 1744140625 ; total 58 ; last 8) - stat 169673 (time 1744140636 ; total 69 ; last 11) - stat 170000 (time 1744140636 ; total 69 ; last 0) - stat 180000 (time 1744140645 ; total 78 ; last 9) - stat 190000 (time 1744140655 ; total 88 ; last 10) - stat 200000 (time 1744140665 ; total 98 ; last 10) total: 202190 stats in 100 seconds: 2021.900024 stats/second - unlinked 0 (time 1744140617 ; total 0 ; last 0) - unlinked 10000 (time 1744140676 ; total 59 ; last 59) unlink(/mnt/lustre2/lockdir/lockfile17991) error: No such file or directory total: 17990 unlinks in 91 seconds: 197.692307 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (23650) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (23653) - No such process PASS 12 (143s) == sanityn test 13: test directory page revocation ======= 15:31:49 (1744140709) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d13 f13 f13 PASS 13 (2s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 15:31:51 (1744140711) striped dir -i0 -c2 -H all_char /mnt/lustre/d14aa.sanityn multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7933 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 336: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4930: 25068 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (2s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 15:31:53 (1744140713) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d14ab.sanityn open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4930: 25715 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (2s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 15:31:55 (1744140715) striped dir -i0 -c2 -H crush /mnt/lustre/d14b.sanityn truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 351: 26359 Terminated $DIR1/$tdir/sleep 60 PASS 14b (2s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 15:31:57 (1744140717) striped dir -i0 -c2 -H crush /mnt/lustre/d14c.sanityn cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 363: 27006 Terminated $DIR1/$tdir/sleep 60 PASS 14c (1s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 15:31:59 (1744140719) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d14d.sanityn chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 374: 27652 Terminated $DIR1/$tdir/sleep 60 PASS 14d (2s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 15:32:01 (1744140721) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_CODE=117440512 CLIENT_OS_VERSION_ID=7 STRIPECOUNT=2 ORIGFREE=7207252 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7207252kB free gt MAXFREE 800000kB, increase 800000 (or reduce test fs size) to proceed Waiting for MDT destroys to complete SKIP 15 (7s) == sanityn test 16a: 2500 iterations of dual-mount fsx === 15:32:09 (1744140729) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.154814 s, 54.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.135287 s, 62.0 MB/s Chance of close/open is 1 in 50 Seed set to 730 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x8eb069 fallocating to largest ever: 0x94708e 000100[1] 1744140732.282326 mapwrite 0x183cdd thru 0x1868f3 (0x2c17 bytes) 000200[0] 1744140733.738906 trunc from 0x4d19f8 to 0x239624 (0x2983d3 bytes) truncating to largest ever: 0x9811ce truncating to largest ever: 0x9bbbe8 000400[1] 1744140736.817728 trunc from 0x242241 to 0x0cd76b (0x174ad5 bytes) 000500[1] 1744140737.876443 fallocate 0x6fc2c3 thru 0x6fcbaa (0x8e8 bytes) truncating to largest ever: 0x9c77cd truncating to largest ever: 0x9e1c78 000700[0] 1744140740.376360 mapwrite 0x166467 thru 0x1739aa (0xd544 bytes) 000800[0] 1744140741.749806 write 0x1536d1 thru 0x162551 (0xee81 bytes) 000900[1] 1744140742.990000 mapwrite 0x0489ea thru 0x04fe47 (0x745e bytes) 001000[0] 1744140744.476850 punch from 0x57ca18 to 0x5854d1 (0x8aba bytes) truncating to largest ever: 0x9ec20e 001200[0] 1744140746.833495 write 0x94edab thru 0x95727f (0x84d5 bytes) 001500[0] 1744140750.811621 mapwrite 0x380f4c thru 0x385d38 (0x4ded bytes) 001700[0] 1744140753.260280 mapread 0x29535e thru 0x298911 (0x35b4 bytes) fallocating to largest ever: 0xa00000 001800[1] 1744140754.500045 punch from 0x3fd543 to 0x405f4c (0x8a0a bytes) 001900[1] 1744140755.680004 read 0x1e89a2 thru 0x1f52ea (0xc949 bytes) 002000[1] 1744140757.054142 fallocate 0x70d241 thru 0x7160ff (0x8ebf bytes) 002200[1] 1744140759.068782 read 0x17c62f thru 0x186b0d (0xa4df bytes) 002300[1] 1744140760.192176 read 0x14b79f thru 0x159240 (0xdaa2 bytes) 002500[1] 1744140762.767248 punch from 0x0b10bb to 0x0bf950 (0xe896 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 762 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x6a9b7a truncating to largest ever: 0x7a11a6 fallocating to largest ever: 0x8153b7 truncating to largest ever: 0x90e9e8 fallocating to largest ever: 0x9417fe fallocating to largest ever: 0x98745b truncating to largest ever: 0x9d24d8 truncating to largest ever: 0x9d46c9 000400[1] 1744140766.888721 mapwrite 0x120000 thru 0x12c9cf (0xc9d0 bytes) 000500[1] 1744140767.745675 mapwrite 0x45c000 thru 0x4679c9 (0xb9ca bytes) 000600[1] 1744140768.625174 punch from 0x3c777e to 0x3d22c2 (0xab45 bytes) 000800[1] 1744140770.558949 mapwrite 0x36e000 thru 0x375362 (0x7363 bytes) fallocating to largest ever: 0x9ec299 001000[1] 1744140772.278551 read 0x5c8000 thru 0x5c839e (0x39f bytes) 001100[0] 1744140773.174741 punch from 0x2fa06a to 0x307c92 (0xdc29 bytes) 001300[1] 1744140775.155798 write 0x190000 thru 0x19d03c (0xd03d bytes) 001400[0] 1744140775.967419 trunc from 0x897158 to 0x036394 (0x860dc3 bytes) fallocating to largest ever: 0x9fb9ad 001700[1] 1744140778.922013 fallocate 0x6f961e thru 0x704442 (0xae25 bytes) 001800[0] 1744140779.877817 fallocate 0x095907 thru 0x09cff7 (0x76f1 bytes) 002000[0] 1744140782.071062 read_OD 0x1b2000 thru 0x1b8fff (0x7000 bytes) fallocating to largest ever: 0xa00000 002100[0] 1744140782.897644 write_OD 0x95e000 thru 0x96cfff (0xf000 bytes) 002200[1] 1744140784.209395 read 0x4ff000 thru 0x5046cb (0x56cc bytes) 002300[0] 1744140785.238104 read_OD 0x8b0000 thru 0x8bafff (0xb000 bytes) 002400[1] 1744140786.406570 write 0x848000 thru 0x852080 (0xa081 bytes) All operations completed A-OK! PASS 16a (59s) == sanityn test 16b: 2500 iterations of dual-mount fsx at small size ========================================================== 15:33:08 (1744140788) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.136156 s, 61.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.128297 s, 65.4 MB/s Chance of close/open is 1 in 50 Seed set to 789 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x8dd287 truncating to largest ever: 0x8f840f truncating to largest ever: 0x9e454d fallocating to largest ever: 0x9f478a 000300[0] 1744140793.051072 fallocate 0x8f6e99 thru 0x8f7421 (0x589 bytes) truncating to largest ever: 0x9f9094 000400[1] 1744140794.051798 punch from 0x1302dd to 0x1306c9 (0x3ed bytes) 000600[1] 1744140796.421795 trunc from 0x9932c5 to 0x95011f (0x431a5 bytes) 000800[0] 1744140798.556341 punch from 0x05fade to 0x05fe9b (0x3be bytes) 000900[1] 1744140799.698304 mapread 0x291fc3 thru 0x29227c (0x2ba bytes) 001000[1] 1744140800.803321 trunc from 0x5aada7 to 0x4577d1 (0x1535d5 bytes) 001100[0] 1744140802.385100 mapwrite 0x1bed5e thru 0x1bf9c2 (0xc65 bytes) 001200[0] 1744140803.600588 fallocate 0x2744ef thru 0x275f1c (0x1a2e bytes) fallocating to largest ever: 0x9fa31b 001300[0] 1744140804.700567 mapwrite 0x1b905e thru 0x1baa00 (0x19a3 bytes) 001400[0] 1744140805.699453 punch from 0x2b67af to 0x2b73ce (0xc20 bytes) 001500[0] 1744140806.474656 mapwrite 0x29fa0e thru 0x2a16b5 (0x1ca8 bytes) 001600[1] 1744140807.244988 fallocate 0x65512c thru 0x655e57 (0xd2c bytes) 001700[0] 1744140808.117953 write 0x6a02a7 thru 0x6a0bb6 (0x910 bytes) 001800[1] 1744140808.994403 mapwrite 0x02a893 thru 0x02be0e (0x157c bytes) 002000[1] 1744140811.325837 fallocate 0x8c9deb thru 0x8c9f17 (0x12d bytes) 002100[1] 1744140812.396946 read 0x044cbb thru 0x046527 (0x186d bytes) 002200[1] 1744140813.290543 mapread 0x004c7b thru 0x00612a (0x14b0 bytes) 002300[1] 1744140814.146085 read 0x2ed5fc thru 0x2ef55d (0x1f62 bytes) 002400[0] 1744140815.281698 mapwrite 0x3573b2 thru 0x357b77 (0x7c6 bytes) 002500[0] 1744140816.201522 write 0x16865b thru 0x16a197 (0x1b3d bytes) All operations completed A-OK! PASS 16b (29s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 15:33:37 (1744140817) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.117047 s, 71.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.107507 s, 78.0 MB/s Waiting for MDT destroys to complete Chance of close/open is 1 in 50 Seed set to 826 fd 0: /mnt/lustre/f16c.sanityn fd 1: /mnt/lustre2/f16c.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read skipping zero length punch hole skipping zero length punch hole skipping zero length punch hole truncating to largest ever: 0x4d994 truncating to largest ever: 0x5291ee fallocating to largest ever: 0x8e10da 000200[1] 1744140828.563899 mapwrite 0x165cc5 thru 0x16a17e (0x44ba bytes) fallocating to largest ever: 0x900ba3 fallocating to largest ever: 0x963ae4 000300[0] 1744140829.465741 fallocate 0x7bd3e4 thru 0x7c22cd (0x4eea bytes) truncating to largest ever: 0x9e9f13 000400[0] 1744140830.267752 fallocate 0x15445d thru 0x15a1e1 (0x5d85 bytes) 000500[1] 1744140831.176887 punch from 0x32a211 to 0x335211 (0xb001 bytes) 000600[0] 1744140832.003335 fallocate 0x7e580b thru 0x7f3ab4 (0xe2aa bytes) 000700[1] 1744140832.792527 fallocate 0x1b8e74 thru 0x1bb469 (0x25f6 bytes) fallocating to largest ever: 0x9f2b13 000800[0] 1744140833.604111 trunc from 0x63a55f to 0x5e807b (0x524e3 bytes) 000900[1] 1744140834.357388 trunc from 0x9b15cf to 0x9d7853 (0x26285 bytes) 001100[1] 1744140835.663854 punch from 0x356162 to 0x362f56 (0xcdf5 bytes) 001200[0] 1744140836.408802 mapwrite 0x718d0a thru 0x71ea52 (0x5d49 bytes) 001300[1] 1744140837.058775 mapwrite 0x8c083d thru 0x8cb153 (0xa917 bytes) fallocating to largest ever: 0x9f55aa 001400[1] 1744140837.715207 write 0x64ffe2 thru 0x656f3a (0x6f59 bytes) 001500[1] 1744140838.521842 write 0x826445 thru 0x82b028 (0x4be4 bytes) 001600[1] 1744140839.125843 write 0x57b45f thru 0x58a675 (0xf217 bytes) 001700[0] 1744140839.862056 mapread 0x64ee98 thru 0x65725a (0x83c3 bytes) 001900[1] 1744140841.680547 write 0x9d5e04 thru 0x9dd181 (0x737e bytes) 002000[1] 1744140842.515421 write 0x6991cb thru 0x69b313 (0x2149 bytes) 002100[1] 1744140843.293677 mapread 0x819132 thru 0x822dfc (0x9ccb bytes) 002200[1] 1744140844.174992 mapread 0x03207d thru 0x041ea3 (0xfe27 bytes) 002300[1] 1744140845.161690 punch from 0x6dabf6 to 0x6dd6fd (0x2b08 bytes) 002400[0] 1744140845.981699 punch from 0x0df21a to 0x0e0929 (0x1710 bytes) 002500[1] 1744140846.740390 mapread 0x19b60b thru 0x1a04ed (0x4ee3 bytes) All operations completed A-OK! PASS 16c (31s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 15:34:08 (1744140848) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.114352 s, 73.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.103257 s, 81.2 MB/s ldlm.namespaces.MGC192.168.202.113@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a9803000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff88012a6ad800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a9803000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff88012a6ad800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a9803000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a9803000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 1.77334 s, 59.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.664784 s, 158 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.698933 s, 150 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 1.25436 s, 83.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.846371 s, 124 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.844998 s, 124 MB/s PASS 16d (9s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 15:34:17 (1744140857) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0699789 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112904 s, 92.9 MB/s PASS 16e (1s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 15:34:18 (1744140858) TIMEOUT PASS 16f (22s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 15:34:40 (1744140880) TIMEOUT PASS 16g (22s) == sanityn test 16h: mmap read after truncate file ======= 15:35:02 (1744140902) striped dir -i0 -c2 -H crush2 /mnt/lustre/d16h.sanityn ------- before truncate -------- 0003220 3035 3a31 3035 3a31 2f3a 6f68 656d 732f 0003240 6e61 7469 7579 7273 3a31 622f 6e69 622f 0003260 7361 0a68 0003264 ------- after truncate -------- 0001460 7373 6468 783a 373a 3a34 3437 503a 6972 0001500 6976 656c 6567 732d 7065 7261 7461 6465 0001520 5320 4853 2f3a 6176 2f72 0001532 PASS 16h (1s) == sanityn test 16i: read after truncate file ============ 15:35:03 (1744140903) striped dir -i0 -c2 -H crush /mnt/lustre/d16i.sanityn ------- before truncate -------- 0003220 3035 3a31 3035 3a31 2f3a 6f68 656d 732f 0003240 6e61 7469 7579 7273 3a31 622f 6e69 622f 0003260 7361 0a68 0003264 ------- after truncate -------- 0001460 7373 6468 783a 373a 3a34 3437 503a 6972 0001500 6976 656c 6567 732d 7065 7261 7461 6465 0001520 5320 4853 2f3a 6176 2f72 0001532 PASS 16i (2s) == sanityn test 16j: race dio with buffered i/o ========== 15:35:05 (1744140905) 1+0 records in 1+0 records out 26214400 bytes (26 MB) copied, 0.362401 s, 72.3 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.315374 s, 83.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.340177 s, 77.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.182649 s, 144 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.297329 s, 88.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.227416 s, 115 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.353131 s, 74.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied26214400 bytes (26 MB) copied, 0.330424 s, 79.3 MB/s , 0.331443 s, 79.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.355285 s, 73.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.464756 s, 56.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.321096 s, 81.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.335814 s, 78.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.431945 s, 60.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.509208 s, 51.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.345685 s, 75.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.366219 s, 71.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.38155 s, 68.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.440814 s, 59.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (7s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 15:35:12 (1744140912) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fallocating to largest ever: 0x3428364 skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fallocating to largest ever: 0x3428364 skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fallocating to largest ever: 0x3428364 fallocating to largest ever: 0x3428364 skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fallocating to largest ever: 0x3428364 skipping zero size read truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db fallocating to largest ever: 0x3f1a2be fallocating to largest ever: 0x3f1a2be fallocating to largest ever: 0x3f1a2be fallocating to largest ever: 0x3f1a2be fallocating to largest ever: 0x3f1a2be fallocating to largest ever: 0x3f8dc08 fallocating to largest ever: 0x3f8dc08 fallocating to largest ever: 0x3f8dc08 fallocating to largest ever: 0x3f8dc08 fallocating to largest ever: 0x3f8dc08 All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Tue Apr 8 15:35:17 EDT 2025 + fsx 2 done: rc=0 Tue Apr 8 15:35:17 EDT 2025 + fsx 3 done: rc=0 Tue Apr 8 15:35:17 EDT 2025 + fsx 4 done: rc=0 Tue Apr 8 15:35:17 EDT 2025 + fsx 5 done: rc=0 Tue Apr 8 15:35:17 EDT 2025 PASS 16k (9s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 15:35:21 (1744140921) fail_loc=0x8000030a PASS 17 (4s) == sanityn test 18: mmap sanity check =========================================================================================== 15:35:25 (1744140925) mmap test1: basic mmap operation (PASS, 0.021259s) mmap test2: MAP_PRIVATE not write back (PASS, 0.012552s) mmap test3: concurrent mmap ops on two nodes (PASS, 2.3901s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.0773s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.028478s) mmap test6: check mmap write/read content on two nodes (PASS, 0.027881s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.30524s) mmap test11: multi-thread shared mmap (PASS, 0.31932s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0113s) PASS 18 (11s) == sanityn test 19: test concurrent uncached read races ========================================================================= 15:35:36 (1744140936) 32+0 records in 32+0 records out 16777216 bytes (17 MB) copied, 0.137583 s, 122 MB/s loop 5 loop 10 loop 15 loop 20 PASS 19 (7s) == sanityn test 20: test extra readahead page left in cache ============================================================== 15:35:43 (1744140943) striped dir -i0 -c2 -H crush2 /mnt/lustre/d20.sanityn PASS 20 (1s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 15:35:44 (1744140944) striped dir -i1 -c2 -H crush /mnt/lustre/d21.sanityn rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (2s) == sanityn test 23: others should see updated atime while another read============================================================== 15:35:46 (1744140946) now is 1744140946 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7933 new atime is 1744141007 PASS 23 (63s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 15:36:49 (1744141009) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 33516 3573504 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5008 3602012 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38524 7175516 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2060 1285628 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 33516 3573504 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 5008 3602012 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 38524 7175516 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 352 999.7K 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1000.0K 277 999.7K 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 256.0K 405 255.6K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 256.0K 374 255.6K 1% /mnt/lustre[OST:1] filesystem_summary: 511.9K 629 511.2K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 352 999.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1000.0K 277 999.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 256.0K 405 255.6K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 256.0K 374 255.6K 1% /mnt/lustre2[OST:1] filesystem_summary: 511.9K 629 511.2K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 1.3G 2.0M 1.2G 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1.3G 1.8M 1.2G 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3.7G 32.7M 3.4G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.7G 4.9M 3.4G 1% /mnt/lustre[OST:1] filesystem_summary: 7.3G 37.6M 6.8G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 352 1023648 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1024000 277 1023723 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 262144 405 261739 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 262144 374 261770 1% /mnt/lustre2[OST:1] filesystem_summary: 524138 629 523509 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 33516 3573504 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5008 3602012 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38524 7175516 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 352 999.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1000.0K 277 999.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 256.0K 405 255.6K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 256.0K 374 255.6K 1% /mnt/lustre2[OST:1] filesystem_summary: 511.9K 629 511.2K 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 352 1023648 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1024000 277 1023723 1% /mnt/lustre[MDT:1] filesystem_summary: 2048000 629 2047371 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 352 1023648 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1024000 277 1023723 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 262144 405 261739 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 262144 374 261770 1% /mnt/lustre2[OST:1] filesystem_summary: 524138 629 523509 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 33516 3573504 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5008 3602012 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38524 7175516 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2060 1285628 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 33516 3573504 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 5008 3602012 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 38524 7175516 1% /mnt/lustre2 PASS 24a (1s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 15:36:50 (1744141010) PASS 24b (1s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 15:36:52 (1744141012) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (1s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 15:36:53 (1744141013) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] PASS 25b (2s) == sanityn test 26a: allow mtime to get older ============ 15:36:55 (1744141015) utime: good mknod times 1744141014 <= 1744141015 <= 1744141015 for /mnt/lustre/f26a utime: good mknod times 1744141014 <= 1744141015 <= 1744141015 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (2s) == sanityn test 26b: sync mtime between ost and mds ====== 15:36:57 (1744141017) PASS 26b (3s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 15:37:00 (1744141020) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7933 File: '/mnt/lustre/f26c.sanityn' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279611 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-04-08 15:37:01.000000000 -0400 Birth: - File: '/mnt/lustre2/f26c.sanityn' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279611 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-04-08 15:37:01.000000000 -0400 Birth: - PASS 26c (3s) SKIP: sanityn test_27 skipping excluded test 27 SKIP: sanityn test_28 skipping ALWAYS excluded test 28 == sanityn test 30: recreate file race =================== 15:37:04 (1744141024) striped dir -i0 -c2 -H crush2 /mnt/lustre/d30.sanityn 0 PASS 30 (3s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 15:37:07 (1744141027) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d31a.sanityn fail_loc=0x314 PASS 31a (3s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 15:37:10 (1744141030) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg213-server mds-ost sync done. Waiting for MDT destroys to complete striped dir -i1 -c2 -H crush2 /mnt/lustre/d31b.sanityn fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.113@tcp:/lustre 7666232 38524 7174520 1% /mnt/lustre2 PASS 31b (7s) == sanityn test 31r: open-rename(replace) race =========== 15:37:17 (1744141037) -rw-r--r-- 1 root root 0 Apr 8 15:37 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (4s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 15:37:21 (1744141041) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1213: 18088 Terminated tail -f $tfile PASS 31s (2s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 15:37:23 (1744141043) PASS 31t (3s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 15:37:26 (1744141046) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (1s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 15:37:27 (1744141047) Stopping /mnt/lustre-mds1 (opts:) on oleg213-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0000 create file on 2nd stripe under striped directory /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre/d33c.sanityn/striped touch /mnt/lustre2/d33c.sanityn/striped/subfile SoLC count 2 Below operations shouldn't trigger Solc: create file on 2nd stripe after setattr chmod 777 /mnt/lustre/d33c.sanityn/striped; touch /mnt/lustre2/d33c.sanityn/striped/subfile create remote dir and local dir /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote create local dir after remote dir creation transaction commit /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote.3; do_facet mds2 /home/green/git/lustre-release/lustre/utils/lctl set_param -n osd*.*MDT0001.force_sync 1; mkdir /mnt/lustre/d33c.sanityn/local.3 Solc count 0 PASS 33c (15s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 15:37:42 (1744141062) remote directory create /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre2/d33d.sanityn/subdir CoS count 2 cross-MDT rename mkdir /mnt/lustre/d33d.sanityn; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn/d1; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33d.sanityn/d2; touch /mnt/lustre/d33d.sanityn/d1/src mv /mnt/lustre2/d33d.sanityn/d1/src /mnt/lustre2/d33d.sanityn/d2/tgt CoS count 4 migrate /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs migrate -m 1 /mnt/lustre2/d33d.sanityn CoS count 2 PASS 33d (10s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 15:37:52 (1744141072) plain dir creation mkdir /mnt/lustre2/d33e.sanityn/plain open file and write echo abc > /mnt/lustre2/d33e.sanityn/f33e.sanityn append write echo dfg >> /mnt/lustre2/d33e.sanityn/f33e.sanityn setattr touch /mnt/lustre2/d33e.sanityn/f33e.sanityn file unlink rm /mnt/lustre2/d33e.sanityn/f33e.sanityn plain dir unlink rmdir /mnt/lustre2/d33e.sanityn/plain striped directory creation /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre2/d33e.sanityn/striped set default LMV to create striped subdir /home/green/git/lustre-release/lustre/utils/lfs setdirstripe -D -c 2 /mnt/lustre/d33e.sanityn striped subdir creation createmany -d /mnt/lustre/d33e.sanityn/subdir 100 total: 100 mkdir in 0.28 seconds: 354.21 ops/second sub file creation and write createmany -o /mnt/lustre/d33e.sanityn/subfile 100; echo abc > /mnt/lustre/d33e.sanityn/subfile1 total: 100 open/close in 0.22 seconds: 457.16 ops/second sub file append write echo dfg >> /mnt/lustre2/d33e.sanityn/subfile2 subdir setatttr touch /mnt/lustre2/d33e.sanityn/subdir1 subdir unlink unlinkmany -d /mnt/lustre/d33e.sanityn/subdir 100 - unlinked 0 (time 1744141074 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second sub file unlink unlinkmany /mnt/lustre2/d33e.sanityn/subfile 100 - unlinked 0 (time 1744141075 ; total 0 ; last 0) total: 100 unlinks in 0 seconds: inf unlinks/second sub file creation follows striped dir chmod chmod 777 /mnt/lustre/d33e.sanityn/striped; touch /mnt/lustre/d33e.sanityn/striped/subfile striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn/striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn CoS count 0 PASS 33e (5s) == sanityn test 34: no lock timeout under IO ============= 15:37:57 (1744141077) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg213-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800a9803000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a9803000.ost_server_uuid in IDLE state after 0 sec oleg213-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800a9803000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800a9803000.ost_server_uuid in FULL state after 0 sec oleg213-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800a9803000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a9803000.ost_server_uuid in IDLE state after 0 sec oleg213-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800a9803000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800a9803000.ost_server_uuid in FULL state after 0 sec oleg213-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800a9803000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a9803000.ost_server_uuid in IDLE state after 0 sec oleg213-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800a9803000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800a9803000.ost_server_uuid in FULL state after 0 sec PASS 34 (82s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 15:39:19 (1744141159) striped dir -i1 -c2 -H crush2 /mnt/lustre/d35.sanityn fail_loc=0x80000317 Race attempt 0 Wait for 28681 28820 for 60 sec... ls: closing directory /mnt/lustre/d35.sanityn: Interrupted system call total: 4000 open/close in 8.25 seconds: 484.58 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.202.113@tcp:/lustre 7.4G 39M 6.9G 1% /mnt/lustre 192.168.202.113@tcp:/lustre 7.4G 39M 6.9G 1% /mnt/lustre2 PASS 35 (63s) == sanityn test 36: handle ESTALE/open-unlink correctly == 15:40:22 (1744141222) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d36.sanityn Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg213-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.32958 s, 159 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7933 Waiting for MDT destroys to complete *** cycle(0) *** before(7174512) after_dd(7123312) after(7174512) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.31466 s, 167 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7933 Waiting for MDT destroys to complete *** cycle(1) *** before(7174512) after_dd(7123312) after(7174512) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.355793 s, 147 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7933 Waiting for MDT destroys to complete *** cycle(2) *** before(7174512) after_dd(7123312) after(7174512) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.310308 s, 169 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7933 Waiting for MDT destroys to complete *** cycle(3) *** before(7174512) after_dd(7123312) after(7174512) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.317098 s, 165 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7933 Waiting for MDT destroys to complete *** cycle(4) *** before(7174512) after_dd(7123312) after(7174512) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.335505 s, 156 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7933 Waiting for MDT destroys to complete *** cycle(5) *** before(7174512) after_dd(7123312) after(7174512) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.405598 s, 129 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7933 Waiting for MDT destroys to complete *** cycle(6) *** before(7174512) after_dd(7123312) after(7174512) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.393711 s, 133 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7933 Waiting for MDT destroys to complete *** cycle(7) *** before(7174512) after_dd(7123312) after(7174512) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.369628 s, 142 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7933 Waiting for MDT destroys to complete *** cycle(8) *** before(7174512) after_dd(7123312) after(7174512) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.305046 s, 172 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7933 Waiting for MDT destroys to complete *** cycle(9) *** before(7174512) after_dd(7123312) after(7174512) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.338812 s, 155 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7933 Waiting for MDT destroys to complete *** cycle(10) *** before(7174512) after_dd(7123312) after(7174512) PASS 36 (171s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 15:43:13 (1744141393) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d37.sanityn multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7933 total: 10000 create in 9.78 seconds: 1022.86 ops/second PASS 37 (12s) == sanityn test 39a: file mtime does not change after rename ========================================================== 15:43:25 (1744141405) repeat after cancel_lru_locks PASS 39a (2s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 15:43:27 (1744141407) repeat after cancel_lru_locks PASS 39b (2s) == sanityn test 39c: check truncate mtime update ================================================================================ 15:43:29 (1744141409) repeat after cancel_lru_locks PASS 39c (3s) == sanityn test 39d: sync write should update mtime ====== 15:43:32 (1744141412) fail_loc=0x411 fail_loc=0 PASS 39d (1s) == sanityn test 40a: pdirops: create vs others ======================================================================== 15:43:33 (1744141413) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=7 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40a (8s) == sanityn test 40b: pdirops: open|create and others ======================================================================== 15:43:41 (1744141421) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=17 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (7s) == sanityn test 40c: pdirops: link and others ======================================================================== 15:43:48 (1744141428) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (7s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 15:43:55 (1744141435) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (7s) == sanityn test 40e: pdirops: rename and others ======================================================================== 15:44:02 (1744141442) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=7 No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (7s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 15:44:09 (1744141449) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=9 mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (4s) == sanityn test 41b: pdirops: create vs create ======================================================================== 15:44:13 (1744141453) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 open: File exists Conflict PASS 41b (4s) == sanityn test 41c: pdirops: create vs link ======================================================================== 15:44:17 (1744141457) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (4s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 15:44:21 (1744141461) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41d (5s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 15:44:26 (1744141466) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 41e (4s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 15:44:30 (1744141470) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41f (4s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 15:44:34 (1744141474) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 Conflict PASS 41g (5s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 15:44:39 (1744141479) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 41h (4s) == sanityn test 41i: reint_open: create vs create ======== 15:44:43 (1744141483) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (652s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 15:55:35 (1744142135) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (4s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 15:55:39 (1744142139) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 42b (5s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 15:55:44 (1744142144) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (5s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 15:55:49 (1744142149) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42d (5s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 15:55:54 (1744142154) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (4s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 15:55:58 (1744142158) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 42f (5s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 15:56:03 (1744142163) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 Conflict PASS 42g (5s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 15:56:08 (1744142168) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 42h (5s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 15:56:13 (1744142173) PASS 43a (22s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 15:56:35 (1744142195) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=38 Conflict PASS 43b (5s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 15:56:40 (1744142200) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43c (5s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 15:56:45 (1744142205) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (5s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 15:56:50 (1744142210) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43e (5s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 15:56:55 (1744142215) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=10 mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (5s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 15:57:00 (1744142220) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 stat: cannot stat '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (6s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 15:57:06 (1744142226) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 43h (4s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 15:57:10 (1744142230) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 43i (5s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 15:57:15 (1744142235) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (40s) == sanityn test 43k: unlink vs create ==================== 15:57:55 (1744142275) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (410s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 16:04:45 (1744142685) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=2 mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Tue Apr 8 16:04:49 EDT 2025 PASS 44a (5s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 16:04:50 (1744142690) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 open: File exists Conflict PASS 44b (5s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 16:04:55 (1744142695) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (6s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 16:05:01 (1744142701) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 44d (5s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 16:05:06 (1744142706) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44e (5s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 16:05:11 (1744142711) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=14 Conflict PASS 44f (5s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 16:05:16 (1744142716) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=11 Conflict PASS 44g (6s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 16:05:22 (1744142722) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44h (5s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 16:05:27 (1744142727) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 lfs mkdir: dirstripe error on '/mnt/lustre2/f44i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f44i.sanityn': File exists Conflict PASS 44i (5s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 16:05:32 (1744142732) PASS 45a (43s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 16:06:15 (1744142775) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=17 Conflict PASS 45b (6s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 16:06:21 (1744142781) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45c (5s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 16:06:26 (1744142786) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (6s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 16:06:32 (1744142792) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45e (5s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 16:06:37 (1744142797) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=13 mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (6s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 16:06:43 (1744142803) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 stat: cannot stat '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (5s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 16:06:49 (1744142809) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45h (5s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 16:06:54 (1744142814) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=7 Conflict PASS 45i (5s) == sanityn test 45j: read vs rename ====================== 16:06:59 (1744142819) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (452s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 16:14:31 (1744143271) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (5s) == sanityn test 46b: pdirops: link vs create ======================================================================== 16:14:36 (1744143276) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 46b (6s) == sanityn test 46c: pdirops: link vs link ======================================================================== 16:14:42 (1744143282) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (5s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 16:14:47 (1744143287) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=7 Conflict PASS 46d (5s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 16:14:52 (1744143292) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 Conflict PASS 46e (5s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 16:14:57 (1744143297) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=11 Conflict PASS 46f (5s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 16:15:02 (1744143302) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=11 Conflict PASS 46g (6s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 16:15:08 (1744143308) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46h (5s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 16:15:13 (1744143313) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=7 lfs mkdir: dirstripe error on '/mnt/lustre2/f46i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f46i.sanityn': File exists Conflict PASS 46i (5s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 16:15:18 (1744143318) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 mkdir: cannot create directory '/mnt/lustre2/f47a.sanityn': File exists Conflict PASS 47a (5s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 16:15:23 (1744143323) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 open: File exists Conflict PASS 47b (6s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 16:15:29 (1744143329) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f47c.sanityn' to '/mnt/lustre2/f47c.sanityn-2': File exists Conflict PASS 47c (5s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 16:15:34 (1744143334) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 47d (6s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 16:15:40 (1744143340) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 mv: cannot overwrite directory '/mnt/lustre2/f47e.sanityn' with non-directory Conflict PASS 47e (5s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 16:15:45 (1744143345) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 Conflict PASS 47f (5s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 16:15:50 (1744143350) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 47g (6s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 16:15:56 (1744143356) 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.00405603 s, 2.5 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (7s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 16:16:03 (1744143363) 0+1 records in 0+1 records out 159 bytes (159 B) copied, 0.0070974 s, 22.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3519: kill: (3805) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 16:16:07 (1744143367) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00555641 s, 184 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0187483 s, 55.9 MB/s 3145728 PASS 51b (14s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 16:16:21 (1744143381) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00565121 s, 181 kB/s PASS 51c (7s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 16:16:28 (1744143388) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012879 s, 81.4 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (2s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 16:16:30 (1744143390) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 1317 0x525 0x2c0000401 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 1317 0x525 0x2c0000401 read lease(1) has applied. PASS 51e (4s) == sanityn test 54: rename locking ======================= 16:16:34 (1744143394) llite.lustre-ffff8800a9803000.xattr_cache=1 llite.lustre-ffff88012a6ad800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: '/mnt/lustre/d1/d2' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762674 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-08 16:16:34.000000000 -0400 Modify: 2025-04-08 16:16:34.000000000 -0400 Change: 2025-04-08 16:16:34.000000000 -0400 Birth: - ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: '/mnt/lustre/d1/d2' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281276 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-08 16:16:40.000000000 -0400 Modify: 2025-04-08 16:16:40.000000000 -0400 Change: 2025-04-08 16:16:40.000000000 -0400 Birth: - ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: '/mnt/lustre/d1/d2' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762677 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-08 16:16:45.000000000 -0400 Modify: 2025-04-08 16:16:45.000000000 -0400 Change: 2025-04-08 16:16:45.000000000 -0400 Birth: - open(O_RDWR|O_CREAT): Is a directory File: '/mnt/lustre/d1' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281277 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-08 16:16:45.000000000 -0400 Modify: 2025-04-08 16:16:47.000000000 -0400 Change: 2025-04-08 16:16:47.000000000 -0400 Birth: - ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: '/mnt/lustre/d1' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762678 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-08 16:16:53.000000000 -0400 Modify: 2025-04-08 16:16:55.000000000 -0400 Change: 2025-04-08 16:16:55.000000000 -0400 Birth: - File: '/mnt/lustre/d1/d2' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281279 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-08 16:16:53.000000000 -0400 Modify: 2025-04-08 16:16:55.000000000 -0400 Change: 2025-04-08 16:16:55.000000000 -0400 Birth: - PASS 54 (37s) == sanityn test 55a: rename vs unlink target dir ========= 16:17:11 (1744143431) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (13s) == sanityn test 55b: rename vs unlink source dir ========= 16:17:25 (1744143445) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55b.sanityn/d1/d2' to '/mnt/lustre/d55b.sanityn/d3/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 16:17:34 (1744143454) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (14s) == sanityn test 55d: rename file vs link ================= 16:17:48 (1744143468) fail_loc=0x155 mv: cannot move '/mnt/lustre/d55d.sanityn/f1' to '/mnt/lustre/d55d.sanityn/d55d.sanityn': Is a directory PASS 55d (11s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 16:17:59 (1744143479) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1744143480 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 12.2078 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (10s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 16:18:09 (1744143489) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: read /mnt/lustre/f56b.sanityn@33554431995904+4096 no data: Success llverdev: read /mnt/lustre/f56b.sanityn@33554431995904+4096 no data: Success llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1744143492 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 2.54321 MB/s read offset: 32768000000kB -nan MB/s llverdev: data verified successfully PASS 56b (56s) == sanityn test 60: Verify data_version behaviour ======== 16:19:05 (1744143545) striped dir -i0 -c2 -H crush /mnt/lustre/d60.sanityn PASS 60 (4s) == sanityn test 70a: cd directory && rm directory ======== 16:19:09 (1744143549) PASS 70a (4s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 16:19:13 (1744143553) PASS 70b (2s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 16:19:15 (1744143555) 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00678172 s, 9.7 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00664922 s, 9.9 MB/s File: '/mnt/lustre2/f71a.sanityn' Size: 262144 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205289281302 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-08 16:19:16.000000000 -0400 Modify: 2025-04-08 16:19:16.000000000 -0400 Change: 2025-04-08 16:19:16.000000000 -0400 Birth: - 42042 extent 0 in offset 65536, length 65536 flags (0x80000000): NET extent 1 in offset 196608, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 2, sum of lengths 131072, expected sum 131072 42043 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00344081 s, 19.0 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00370523 s, 17.7 MB/s File: '/mnt/lustre/f71a.sanityn' Size: 262144 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205289281302 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-08 16:19:16.000000000 -0400 Modify: 2025-04-08 16:19:16.000000000 -0400 Change: 2025-04-08 16:19:16.000000000 -0400 Birth: - 42044 extent 0 in offset 65536, length 65536 flags (0x80000000): NET extent 1 in offset 196608, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 2, sum of lengths 131072, expected sum 131072 42043 PASS 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 16:19:18 (1744143558) 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00687679 s, 9.5 MB/s extent 0 in offset 0, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 1, sum of lengths 65536, expected sum 65536 PASS 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 16:19:21 (1744143561) filefrag 1.46.2.wc5 (26-Mar-2022) check last flag for file with 291 extents 290: 296960.. 296963: 591872.. 591875: 4: 0000: last,net,eof check last flag for file with 292 extents 291: 297984.. 297987: 756736.. 756739: 4: 0001: last,net,eof check last flag for file with 293 extents 292: 299008.. 299011: 585728.. 585731: 4: 0000: last,net,eof PASS 71c (27s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 16:19:48 (1744143588) 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.0255672 s, 2.6 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00190418 s, 34.4 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.002096 s, 31.3 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.0593209 s, 1.1 MB/s disk usage: 262144 /mnt/lustre/f71d.sanityn file size: 458752 /mnt/lustre/f71d.sanityn extent 0 in offset 0, length 65536 flags (0x80000000): NET extent 1 in offset 131072, length 65536 flags (0x80000000): NET extent 2 in offset 262144, length 65536 flags (0x80000000): NET extent 3 in offset 393216, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 4, sum of lengths 262144, expected sum 262144 PASS 71d (29s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 16:20:17 (1744143617) llite.lustre-ffff8800a9803000.xattr_cache=1 llite.lustre-ffff88012a6ad800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (5s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 16:20:22 (1744143622) llite.lustre-ffff8800a9803000.xattr_cache=1 llite.lustre-ffff88012a6ad800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (3s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 16:20:26 (1744143626) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00647335 s, 158 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00721151 s, 142 kB/s 18857: get lock2 18858: get lock1 18857: done 18857: get lock1 18858: done 18858: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 18857: cannot get lock1: Resource deadlock avoided fcntl cmd 7 failed: Resource deadlock avoided 18858: cannot get lock2: Resource deadlock avoided 18858: exit rc=0 18857: done 18857: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 16:20:32 (1744143632) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0217263 s, 96.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0147611 s, 71.0 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4108: 19509 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 16:20:39 (1744143639) striped dir -i0 -c2 -H all_char /mnt/lustre/d76.sanityn opening files: ................ closing files: ................ PASS 76 (46s) == sanityn test 77a: check FIFO NRS policy =============== 16:21:25 (1744143685) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.134897 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0163603 s, 64.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0168073 s, 62.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0357323 s, 29.3 MB/s , 0.0408916 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0414043 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0426173 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0543911 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0494983 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0603333 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0621198 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0656148 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0562784 s, 18.6 MB/s , 0.0705938 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0740909 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0660477 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0768022 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0163082 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0163931 s, 64.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0167829 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0176417 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0210069 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0202474 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0252686 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0153473 s, 68.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0157114 s, 66.7 MB/s , 0.0158974 s, 66.0 MB/s 1048576 bytes (1.0 MB) copied, 0.015814 s, 66.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0188923 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0193408 s, 54.2 MB/s , 0.0190645 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0207765 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0245602 s, 42.7 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg213-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77a.sanityn' PASS 77a (3s) == sanityn test 77b: check CRR-N NRS policy ============== 16:21:29 (1744143689) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.113195 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132597 s, 79.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0138783 s, 75.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0149853 s, 70.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0147701 s, 71.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0217787 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118521 s, 88.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0187798 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0200429 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0226141 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.039508 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0399952 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0389467 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0396359 s, 26.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0408382 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0466198 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0464965 s, 22.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0155642 s, 67.4 MB/s , 0.00979334 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0186293 s, 56.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0235756 s, 44.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0053716 s, 195 MB/s 1048576 bytes (1.0 MB) copied, 0.00561236 s, 187 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0159989 s, 65.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00724091 s, 145 MB/s , 0.00159366 s, 658 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00439955 s, 238 MB/s 1048576 bytes (1.0 MB) copied, 0.0154824 s, 67.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00939466 s, 112 MB/s , 0.00172377 s, 608 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00167149 s, 627 MB/s 1048576 bytes (1.0 MB) copied, 0.0057169 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00210275 s, 499 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=1 removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg213-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.138158 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0202856 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106741 s, 98.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0184126 s, 56.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0330742 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0380154 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.030444 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0437938 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0189226 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0290142 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0445418 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0253916 s, 41.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0428241 s, 24.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0511777 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00812926 s, 129 MB/s 1048576 bytes (1.0 MB) copied, 0.0434491 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0434272 s, 24.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0487188 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.029177 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0189008 s, 55.5 MB/s 1048576 bytes (1.0 MB) copied, 0.024814 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00271778 s, 386 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00264623 s, 396 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0282098 s, 37.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0296335 s, 35.4 MB/s , 0.00241152 s, 435 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0240805 s, 43.5 MB/s , 0.0176669 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0207664 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0270515 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0163446 s, 64.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114563 s, 91.5 MB/s 1048576 bytes (1.0 MB) copied, 0.015036 s, 69.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=1 removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg213-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (3s) == sanityn test 77c: check ORR NRS policy ================ 16:21:32 (1744143692) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.137896 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104948 s, 99.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133021 s, 78.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0164256 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0260107 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125147 s, 83.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0135795 s, 77.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100606 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0213381 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0366039 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0244155 s, 42.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0512499 s, 20.5 MB/s , 0.0506165 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0359767 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0453878 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0614562 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0607724 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00901505 s, 116 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00694308 s, 151 MB/s 1048576 bytes (1.0 MB) copied, 0.00706139 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00761509 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00854552 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120258 s, 87.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00640838 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.016347 s, 64.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00786154 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00816245 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00971845 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105589 s, 99.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0117164 s, 89.5 MB/s , 0.0116383 s, 90.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122366 s, 85.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0138093 s, 75.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg213-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.114501 s, 147 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0194253 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0215766 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0363409 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0538577 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0397293 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0557504 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0537416 s, 19.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0596332 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0575798 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0456617 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0481369 s, 21.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0546077 s, 19.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0586999 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0608958 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.062664 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0548494 s, 19.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00894194 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00947592 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00853977 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113223 s, 92.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010886 s, 96.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00973152 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107763 s, 97.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0099868 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122715 s, 85.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0131681 s, 79.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0131028 s, 80.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0152089 s, 68.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0151128 s, 69.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0148204 s, 70.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0136433 s, 76.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0156321 s, 67.1 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg213-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.115017 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132828 s, 78.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0123596 s, 84.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0379594 s, 27.6 MB/s 1048576 bytes (1.0 MB) copied, 0.041296 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0430863 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0467786 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0520069 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0536373 s, 19.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0476312 s, 22.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0491811 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0470219 s, 22.3 MB/s , 0.0475816 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0539364 s, 19.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0560802 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0620252 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0636912 s, 16.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00752687 s, 139 MB/s , 0.0196733 s, 53.3 MB/s , 0.00965993 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0160257 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00175356 s, 598 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00993246 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0220622 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111845 s, 93.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.02361 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0195269 s, 53.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0123488 s, 84.9 MB/s 1048576 bytes (1.0 MB) copied, 0.0129759 s, 80.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0129418 s, 81.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0130902 s, 80.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0134137 s, 78.2 MB/s , 0.0159386 s, 65.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=3 removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg213-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (5s) == sanityn test 77d: check TRR nrs policy ================ 16:21:37 (1744143697) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.13437 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0119489 s, 87.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0173709 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.039406 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0358557 s, 29.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0571951 s, 18.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0558788 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0554197 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0595988 s, 17.6 MB/s 1048576 bytes (1.0 MB) copied, 0.0618986 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0510017 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0528636 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0574151 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0594118 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0633293 s, 16.6 MB/s , 0.0586992 s, 17.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0707158 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0178419 s, 58.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0274173 s, 38.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00264684 s, 396 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0588155 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0632704 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0531625 s, 19.7 MB/s , 0.0279153 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00165924 s, 632 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0393572 s, 26.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0246318 s, 42.6 MB/s , 0.0363632 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0192094 s, 54.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0124972 s, 83.9 MB/s , 0.00947425 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00710105 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0189454 s, 55.3 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg213-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.129521 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0171178 s, 61.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0272186 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.040972 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0269014 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0439641 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.053639 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0490809 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0584647 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0372716 s, 28.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0386038 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0707984 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0578 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0783649 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0555252 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0611815 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.071161 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0141496 s, 74.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011483 s, 91.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00188205 s, 557 MB/s , 0.00909399 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0054252 s, 193 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00383252 s, 274 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0160481 s, 65.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00164826 s, 636 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00158072 s, 663 MB/s , 0.0121229 s, 86.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00149646 s, 701 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0020939 s, 501 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0231115 s, 45.4 MB/s , 0.00941251 s, 111 MB/s , 0.00737242 s, 142 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00170834 s, 614 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg213-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.12307 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0150507 s, 69.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0160634 s, 65.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0187196 s, 56.0 MB/s 1048576 bytes (1.0 MB) copied, 0.021235 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0160898 s, 65.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132539 s, 79.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0347749 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0173791 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0305148 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0561831 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0602113 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0605035 s, 17.3 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=2 1048576 bytes (1.0 MB) copied, 0.0338968 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0619103 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0641342 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0407802 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00250056 s, 419 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0414658 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0466072 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0329269 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00187746 s, 559 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0404397 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0387208 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0301042 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0262312 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0230212 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0217149 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0202228 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0171795 s, 61.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0193738 s, 54.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.012747 s, 82.3 MB/s , 0.00891502 s, 118 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg213-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (5s) == sanityn test 77e: check TBF NID nrs policy ============ 16:21:42 (1744143702) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.202.13@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.122843 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0207466 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0265157 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0329458 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0323883 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.044852 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0562887 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0562214 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0587342 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0568118 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0630683 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0576626 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0754645 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0776294 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0588313 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0794903 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0617491 s, 17.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0220087 s, 47.6 MB/s , 0.0174032 s, 60.3 MB/s , 0.0120006 s, 87.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0109445 s, 95.8 MB/s , 0.0133109 s, 78.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0137419 s, 76.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0201242 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011011 s, 95.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0162935 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0127173 s, 82.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0128943 s, 81.3 MB/s , 0.0129576 s, 80.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0142981 s, 73.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0145619 s, 72.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0168434 s, 62.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0285834 s, 36.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=4 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg213-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.122836 s, 137 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0145404 s, 72.1 MB/s 1048576 bytes (1.0 MB) copied, 0.0214195 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.016434 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0209953 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0331108 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copiedldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0397811 s, 26.4 MB/s , 0.0408559 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0554675 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0439976 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0699541 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0822032 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0686211 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0913072 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0695333 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0835567 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.101641 s, 10.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108841 s, 96.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00720243 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00161842 s, 648 MB/s 1048576 bytes (1.0 MB) copied, 0.0016344 s, 642 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0139513 s, 75.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0123363 s, 85.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113379 s, 92.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0130834 s, 80.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00151316 s, 693 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0176337 s, 59.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00998612 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00170654 s, 614 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00163528 s, 641 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0164205 s, 63.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0172005 s, 61.0 MB/s , 0.00250358 s, 419 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg213-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.144299 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.019675 s, 53.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0199767 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.033077 s, 31.7 MB/s 1048576 bytes (1.0 MB) copied, 0.0380746 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0498658 s, 21.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0613507 s, 17.1 MB/s , 0.057913 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.061761 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0521357 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0588171 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0630361 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0623975 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.055201 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0620226 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0592759 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0684667 s, 15.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0162874 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0169485 s, 61.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0203436 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0210155 s, 49.9 MB/s , 0.0197003 s, 53.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0171296 s, 61.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0190789 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0191829 s, 54.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0224313 s, 46.7 MB/s , 0.0225234 s, 46.6 MB/s , 0.021976 s, 47.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0273803 s, 38.3 MB/s , 0.0272018 s, 38.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0291036 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0253647 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0305761 s, 34.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg213-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.152294 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0156427 s, 67.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0159421 s, 65.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124221 s, 84.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0483926 s, 21.7 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0592191 s, 17.7 MB/s , 0.0611569 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0603145 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0559444 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0569278 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0559291 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0538751 s, 19.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0640189 s, 16.4 MB/s 1048576 bytes (1.0 MB) copied, 0.065736 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0675063 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0609329 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copiedldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=7 , 0.0672101 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00803591 s, 130 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00673509 s, 156 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out , 0.0135524 s, 77.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0123695 s, 84.8 MB/s 1048576 bytes (1.0 MB) copied, 0.00976639 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101047 s, 104 MB/s , 0.0144891 s, 72.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00941609 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116109 s, 90.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115291 s, 91.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012027 s, 87.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121498 s, 86.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0142565 s, 73.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0149935 s, 69.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0152613 s, 68.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0162667 s, 64.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg213-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' PASS 77e (7s) == sanityn test 77f: check TBF JobID nrs policy ========== 16:21:49 (1744143709) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg213-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.161219 s, 104 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0148772 s, 70.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0142771 s, 73.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0158103 s, 66.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0771835 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.073482 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0799006 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0820733 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0965851 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.103245 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.107104 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.106986 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.111781 s, 9.4 MB/s 1048576 bytes (1.0 MB) copied, 0.109888 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.100886 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0993144 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.108993 s, 9.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=6 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.039227 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00524531 s, 200 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0540225 s, 19.4 MB/s , 0.0461453 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00344741 s, 304 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0340025 s, 30.8 MB/s , 0.042084 s, 24.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0302856 s, 34.6 MB/s , 0.00288791 s, 363 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0561821 s, 18.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0440448 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.036451 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0235861 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0166992 s, 62.8 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0128075 s, 81.9 MB/s , 0.0339943 s, 30.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=2 removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg213-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg213-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.158791 s, 106 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0163023 s, 64.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0193056 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0200978 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0249273 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0639923 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0657309 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0692608 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.05866 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0498479 s, 21.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0596209 s, 17.6 MB/s 1048576 bytes (1.0 MB) copied, 0.0489189 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0776163 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0767265 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0768207 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.05907 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0715988 s, 14.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00339153 s, 309 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0580587 s, 18.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0384624 s, 27.3 MB/s , 0.0462225 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0532646 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0456837 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0593281 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0327612 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0286777 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0398314 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0242678 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00733162 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00903529 s, 116 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0167205 s, 62.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0223712 s, 46.9 MB/s , 0.014808 s, 70.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg213-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg213-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.170451 s, 98.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0166799 s, 62.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0196452 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.017543 s, 59.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.020105 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0198566 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0574717 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0805371 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0848402 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0950783 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0716804 s, 14.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.082819 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0801684 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0794941 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.104409 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.109479 s, 9.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.12647 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0162836 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00195714 s, 536 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0166945 s, 62.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00381216 s, 275 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0140286 s, 74.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0186892 s, 56.1 MB/s , 0.0327741 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0194981 s, 53.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0424535 s, 24.7 MB/s 1048576 bytes (1.0 MB) copied, 0.0291715 s, 35.9 MB/s , 0.0177591 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.023683 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0066301 s, 158 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0177063 s, 59.2 MB/s , 0.029378 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0355305 s, 29.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=3 removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg213-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg213-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.143359 s, 117 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0186987 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0221054 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0257818 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0563952 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0524442 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0483357 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0544125 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0699414 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0653392 s, 16.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0693858 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0704218 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0697865 s, 15.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0917131 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0735863 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0888044 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0965407 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0319026 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0356776 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0289341 s, 36.2 MB/s 1048576 bytes (1.0 MB) copied, 0.029119 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0328178 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0337291 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0314611 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.031843 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0280575 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0321408 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0234625 s, 44.7 MB/s , 0.0210669 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0236004 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0214478 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0233724 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0262059 s, 40.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=1 removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg213-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' PASS 77f (8s) == sanityn test 77g: Change TBF type directly ============ 16:21:57 (1744143717) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (3s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 16:22:01 (1744143721) oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg213-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Unknown error 524 oleg213-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Unknown error 524 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 12 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Unknown error 524 oleg213-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Unknown error 524 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 12 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Unknown error 524 oleg213-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Unknown error 524 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 12 PASS 77h (3s) == sanityn test 77i: Change rank of TBF rule ============= 16:22:04 (1744143724) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (10s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 16:22:14 (1744143734) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.148264 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112168 s, 93.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120264 s, 87.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0152147 s, 68.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0450706 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0344671 s, 30.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0340173 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0483397 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0447504 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0563715 s, 18.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0634047 s, 16.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0533597 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0662487 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0652228 s, 16.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0547883 s, 19.1 MB/s , 0.0560129 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0705609 s, 14.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0425603 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00218482 s, 480 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0548758 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0637574 s, 16.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0468214 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0357836 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0474872 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0265792 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0197948 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0484746 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0276581 s, 37.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.015466 s, 67.8 MB/s , 0.0467559 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0167567 s, 62.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116004 s, 90.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0186759 s, 56.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=1 removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg213-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.94147 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.5052 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (42s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 16:22:56 (1744143776) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg213-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg213-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg213-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg213-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg213-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.280006 s, 59.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0187592 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0230066 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0253135 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0345439 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0351153 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0420194 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0474241 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0570648 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0657794 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0716663 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0790441 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0853199 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0881809 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0925993 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0982061 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.167348 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0200439 s, 52.3 MB/s , 0.0188564 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0172257 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.020068 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0207614 s, 50.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0235247 s, 44.6 MB/s , 0.0230352 s, 45.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0261935 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0198108 s, 52.9 MB/s 1048576 bytes (1.0 MB) copied, 0.0242056 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0214368 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0228248 s, 45.9 MB/s , 0.0297043 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0261424 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0271414 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0270425 s, 38.8 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg213-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.6774 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9587 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg213-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg213-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg213-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg213-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg213-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.252358 s, 66.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0123604 s, 84.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125835 s, 83.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0157409 s, 66.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0236679 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0240741 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0253499 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00794438 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0313958 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0247218 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00827213 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00848933 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00730343 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00767027 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00787942 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00983879 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0362953 s, 28.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00880757 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00931662 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00938658 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00935987 s, 112 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112345 s, 93.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00887661 s, 118 MB/s , 0.00438752 s, 239 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0129166 s, 81.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0100825 s, 104 MB/s , 0.0106899 s, 98.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101249 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00906152 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110795 s, 94.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102971 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0137129 s, 76.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0015683 s, 669 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg213-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.8224 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9686 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (115s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 16:24:51 (1744143891) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg213-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg213-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg213-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg213-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg213-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.250114 s, 67.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132097 s, 79.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126164 s, 83.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0146097 s, 71.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0173661 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0157489 s, 66.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0200225 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00849311 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00959174 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102142 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00871188 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00719027 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00769578 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00830943 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00898405 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00786899 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0397064 s, 26.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00986766 s, 106 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00816581 s, 128 MB/s , 0.0104244 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102382 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115909 s, 90.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101607 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0123363 s, 85.0 MB/s , 0.0112476 s, 93.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109073 s, 96.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104119 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106126 s, 98.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116797 s, 89.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102429 s, 102 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0123763 s, 84.7 MB/s , 0.0131646 s, 79.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0146887 s, 71.4 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg213-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.822 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9701 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg213-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg213-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg213-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg213-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg213-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.250785 s, 66.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107328 s, 97.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103336 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.014088 s, 74.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0221918 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0255024 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0322044 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00728731 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00874245 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00845665 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120802 s, 86.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.017015 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0206945 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0289807 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0294666 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0301913 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0622131 s, 16.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00907349 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00888295 s, 118 MB/s , 0.00966697 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00967464 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109235 s, 96.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107355 s, 97.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101044 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00957949 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0098159 s, 107 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101551 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0101967 s, 103 MB/s , 0.0108431 s, 96.7 MB/s , 0.0103333 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00978529 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108828 s, 96.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0139126 s, 75.4 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg213-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.8292 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9403 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (109s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 16:26:40 (1744144000) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg213-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.164851 s, 102 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0200297 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0280417 s, 37.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0326291 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0181985 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0274256 s, 38.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0161666 s, 64.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0381618 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00656216 s, 160 MB/s 1048576 bytes (1.0 MB) copied, 0.0365359 s, 28.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0391738 s, 26.8 MB/s 1048576 bytes (1.0 MB) copied, 0.0551962 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0513526 s, 20.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.043016 s, 24.4 MB/s 1048576 bytes (1.0 MB) copied, 0.059064 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0620951 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.066133 s, 15.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00832748 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0084807 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00726861 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00928169 s, 113 MB/s 1048576 bytes (1.0 MB) copied, 0.0110352 s, 95.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00884251 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00882977 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00307404 s, 341 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102395 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00975567 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0093583 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00939287 s, 112 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00934987 s, 112 MB/s , 0.00876404 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116272 s, 90.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0135018 s, 77.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=1 removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg213-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.90727 s, 21.4 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.7368 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.13@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.13@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0950668 s, 176 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00750055 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0129668 s, 80.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0177618 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0205162 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0400622 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0419769 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0302772 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0367248 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0410525 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0786063 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0775699 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0801046 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0818348 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0872848 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0907706 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0918765 s, 11.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0369959 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0384042 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0398358 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0376695 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.038911 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0390376 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0382906 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0399779 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0402745 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0409069 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0408748 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0420827 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0411135 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0413369 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0412835 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0427937 s, 24.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=1 removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg213-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.89146 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.73299 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.13@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg213-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.10785 s, 156 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00896891 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00822009 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117681 s, 89.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0134818 s, 77.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0164258 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0193899 s, 54.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0342607 s, 30.6 MB/s , 0.0342598 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0327295 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0395776 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0519198 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0516304 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0437705 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0520158 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0449474 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0537558 s, 19.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00939816 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00177793 s, 590 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00831816 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00761433 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00418853 s, 250 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00884406 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00905488 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00905972 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103082 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011507 s, 91.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124892 s, 84.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101633 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.013534 s, 77.5 MB/s , 0.0133766 s, 78.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109917 s, 95.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0129253 s, 81.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=1 removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg213-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.88656 s, 21.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.87531 s, 21.5 MB/s Read runtime is 5 s, speed is 20.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg213-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.103204 s, 163 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00717478 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00879301 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0159807 s, 65.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108092 s, 97.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0296143 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0217482 s, 48.2 MB/s , 0.0261521 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0303547 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0269808 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0684546 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0864337 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0841932 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0761276 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0861802 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0730172 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0920298 s, 11.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00734222 s, 143 MB/s , 0.00748386 s, 140 MB/s , 0.00751905 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0064667 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00845383 s, 124 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105472 s, 99.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00773575 s, 136 MB/s , 0.00846965 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00937252 s, 112 MB/s 1048576 bytes (1.0 MB) copied, 0.0109274 s, 96.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00937803 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00953242 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00904039 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102888 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00995467 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112388 s, 93.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg213-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.7337 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.72525 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.88319 s, 21.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.73215 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg213-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.100255 s, 167 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0096573 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00845941 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0195514 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0265085 s, 39.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0260257 s, 40.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0392234 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.029545 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0312052 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0293947 s, 35.7 MB/s 1048576 bytes (1.0 MB) copied, 0.0310755 s, 33.7 MB/s 1048576 bytes (1.0 MB) copied, 0.0397237 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.036847 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0369045 s, 28.4 MB/s , 0.0426109 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0414736 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0350141 s, 29.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00616153 s, 170 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00980803 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00579445 s, 181 MB/s 1048576 bytes (1.0 MB) copied, 0.00583646 s, 180 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0067475 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00137748 s, 761 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00929701 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00867859 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00899489 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00924796 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00939449 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010765 s, 97.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110953 s, 94.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118926 s, 88.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106523 s, 98.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110017 s, 95.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg213-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.643 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.4438 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg213-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.177529 s, 94.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0157997 s, 66.4 MB/s 1048576 bytes (1.0 MB) copied, 0.017866 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0244436 s, 42.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.044828 s, 23.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0428421 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0491143 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0358528 s, 29.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0338524 s, 31.0 MB/s , 0.0518866 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0440931 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0469701 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0467455 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0477589 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0556765 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0543038 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0558078 s, 18.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00214203 s, 490 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00247584 s, 424 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.021529 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0195066 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0164661 s, 63.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.023775 s, 44.1 MB/s , 0.0212314 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0246376 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00142068 s, 738 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0129205 s, 81.2 MB/s , 0.0164249 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116319 s, 90.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00840073 s, 125 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00400106 s, 262 MB/s 1048576 bytes (1.0 MB) copied, 0.00609058 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0057741 s, 182 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=2 removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg213-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.90605 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.74585 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg213-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.139493 s, 120 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101653 s, 103 MB/s 1048576 bytes (1.0 MB) copied, 0.0101818 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0258267 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0250723 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.029564 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0317723 s, 33.0 MB/s , 0.0340848 s, 30.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.038189 s, 27.5 MB/s 1048576 bytes (1.0 MB) copied, 0.034571 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0398267 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.041713 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0426913 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0373853 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0385957 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0420016 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0473593 s, 22.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00910903 s, 115 MB/s , 0.0100667 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00924367 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108146 s, 97.0 MB/s , 0.00983094 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0123882 s, 84.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00512216 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0119931 s, 87.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121985 s, 86.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0127863 s, 82.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0131604 s, 79.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121854 s, 86.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0139299 s, 75.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0127351 s, 82.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132371 s, 79.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.013645 s, 76.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg213-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.76383 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.74706 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.75488 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.74154 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (263s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 16:31:04 (1744144264) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (3s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 16:31:07 (1744144267) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg213-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 3.29664 s, 318 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg213-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 40.0089 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (51s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 16:31:58 (1744144318) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.12624 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120593 s, 87.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0224137 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0127252 s, 82.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118031 s, 88.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0325673 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0146741 s, 71.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0362719 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132753 s, 79.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0383891 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0371517 s, 28.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0375443 s, 27.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0384463 s, 27.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0415586 s, 25.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0522824 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0557695 s, 18.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0600425 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00727431 s, 144 MB/s 1048576 bytes (1.0 MB) copied, 0.00996764 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00168469 s, 622 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00152493 s, 688 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106666 s, 98.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0155135 s, 67.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0169495 s, 61.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0152687 s, 68.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0188516 s, 55.6 MB/s , 0.00160702 s, 652 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0127486 s, 82.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0130274 s, 80.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00529976 s, 198 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0140285 s, 74.7 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0107173 s, 97.8 MB/s 1048576 bytes (1.0 MB) copied, 0.00487403 s, 215 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=3 removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg213-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.91129 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.97563 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.152618 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0149322 s, 70.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0156331 s, 67.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0314123 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0333508 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0388178 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0410561 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.042714 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0391427 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0347337 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0350692 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0432168 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0503366 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0339467 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0444669 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0467135 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0518161 s, 20.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a6ad800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.105571 s, 9.9 MB/s , 0.105892 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.105558 s, 9.9 MB/s , 0.110574 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.106074 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.110644 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.108841 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.109251 s, 9.6 MB/s 1048576 bytes (1.0 MB) copied, 0.110688 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.110238 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.110741 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.112158 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.113786 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.113037 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.113223 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.116563 s, 9.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012a6ad800.lock_unused_count=1 removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg213-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg213-client.virtnet' removed directory: '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.90845 s, 21.4 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.97329 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (45s) == sanityn test 77o: Changing rank should not panic ====== 16:32:43 (1744144363) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 16:32:47 (1744144367) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (37s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 16:33:24 (1744144404) mds.MDS.mdt.nrs_policies=tbf oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg213-client: oleg213-server: ssh exited with exit code 36 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg213-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg213-client: oleg213-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (13s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 16:33:37 (1744144417) striped dir -i0 -c1 -H crush2 /mnt/lustre/d77r.sanityn mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 18915 kill md_thread xargs PASS 77r (37s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 16:34:14 (1744144454) oleg213-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg213-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg213-client: oleg213-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 16:34:17 (1744144457) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d79.sanityn File: '/mnt/lustre/d79.sanityn' Size: 8192 Blocks: 16 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798763890 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-08 16:34:18.000000000 -0400 Modify: 2025-04-08 16:34:18.000000000 -0400 Change: 2025-04-08 16:34:18.000000000 -0400 Birth: - fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 16:34:30 (1744144470) total: 10 open/close in 0.05 seconds: 183.43 ops/second multiop /mnt/lustre2/d80a.sanityn/f80a.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7933 lfs migrate: /mnt/lustre/d80a.sanityn/f80a.sanityn migrate failed: Device or resource busy (16) Finish migration, then checking.. PASS 80a (5s) == sanityn test 80b: Accessing directory during migration ========================================================== 16:34:35 (1744144475) start migration thread 29660 accessing the migrating directory for 1 minute... concurrent access 1521 failures, 1935 successes /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4930: 29660 Killed ( while true; do mdt_idx=$((RANDOM % MDSCOUNT)); $LFS migrate -m $mdt_idx $migdir1 &>/dev/null; done ) (wd: ~) PASS 80b (61s) == sanityn test 81a: rename and stat under striped directory ========================================================== 16:35:36 (1744144536) stat: cannot stat 'd0/0': No such file or directory File: 'd0/0' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289284070 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-08 16:35:37.000000000 -0400 Modify: 2025-04-08 16:35:37.000000000 -0400 Change: 2025-04-08 16:35:37.000000000 -0400 Birth: - PASS 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 16:35:39 (1744144539) total: 2500 open/close in 9.73 seconds: 256.83 ops/second /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4930: 2757 Killed ( while true; do touch $DIR1/$tdir; done ) (wd: ~) PASS 81b (49s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 16:36:29 (1744144589) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 16:36:30 (1744144590) total: 2500 open/close in 7.91 seconds: 316.17 ops/second total: 2500 open/close in 7.41 seconds: 337.45 ops/second snapshot_time 1744144630.838977434 secs.nsecs start_time 1744144592.058522846 secs.nsecs elapsed_time 38.780454588 secs.nsecs open 2505 samples [usecs] 42 1413 709245 233794247 close 2504 samples [usecs] 37 308 203758 19484092 mknod 2501 samples [usecs] 109 1352 592416 164852366 unlink 2500 samples [usecs] 165 33766 830442 1925591514 mkdir 2 samples [usecs] 638 3003 3641 9425053 rmdir 2 samples [usecs] 1479 3273 4752 12899970 rename 2500 samples [usecs] 162 4762 1880902 2293749916 getattr 3764 samples [usecs] 0 326 60026 1377058 statfs 8 samples [usecs] 8 16 93 1161 parallel_rename_file 2500 samples [usecs] 162 4762 1880902 2293749916 crossdir_rename 2500 samples [usecs] 162 4762 1880902 2293749916 snapshot_time 1744144630.839230661 secs.nsecs start_time 1744144592.058686152 secs.nsecs elapsed_time 38.780544509 secs.nsecs open 2504 samples [usecs] 42 820 672076 206394032 close 2504 samples [usecs] 37 224 194022 17534420 mknod 2500 samples [usecs] 112 792 562081 145075695 unlink 2500 samples [usecs] 165 17352 787470 1018152562 mkdir 2 samples [usecs] 425 3341 3766 11342906 rmdir 2 samples [usecs] 954 7777 8731 61391845 rename 2500 samples [usecs] 168 4188 2779332 5332868888 getattr 3757 samples [usecs] 6 912 72540 2632580 statfs 8 samples [usecs] 6 14 78 828 parallel_rename_file 2500 samples [usecs] 168 4188 2779332 5332868888 crossdir_rename 2500 samples [usecs] 168 4188 2779332 5332868888 PASS 81d (41s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 16:37:11 (1744144631) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7933 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7933 PASS 82 (2s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 16:37:13 (1744144633) start pid 17682 to create/unlink striped directory start pid 17683 to stat striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5553: 17682 Terminated ( cd $DIR1; while true; do $LFS mkdir -i1 -c2 $tdir; rmdir $tdir; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5553: 17683 Terminated ( cd $DIR2; while true; do stat $tdir > /dev/null 2>&1; done ) PASS 83 (122s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 16:39:15 (1744144755) fail_loc=0x8000060b PASS 84 (8s) == sanityn test 85: Lustre API root cache race =========== 16:39:23 (1744144763) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1744144764 Finishing test test1 at 1744144764 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 16:39:26 (1744144766) start pid 31275 to open/create under striped directory start pid 31276 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5620: 31275 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; touch $tdir/f{0..3} > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5620: 31276 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 90 (182s) == sanityn test 91: chmod and unlink striped directory === 16:42:28 (1744144948) start pid 12677 to chmod striped directory start pid 12678 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5657: 12677 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; chmod go+w $tdir > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5657: 12678 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 91 (183s) == sanityn test 92: create remote directory under orphan directory ========================================================== 16:45:31 (1744145131) fail_loc=0x1408 mkdir: cannot create directory '/mnt/lustre2/d92.sanityn/dir': No such file or directory lfs setdirstripe: cannot resolve path '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory (2) lfs setdirstripe: '/mnt/lustre2/d92.sanityn/remote_dir' is not on a Lustre filesystem: No such file or directory (2) lfs setdirstripe: cannot create dir '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory fail_loc=0 PASS 92 (2s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 16:45:33 (1744145133) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6564 0x19a4 0x2c0000400 0 6570 0x19aa 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6771 0x1a73 0x2c0000401 0 6781 0x1a7d 0x280000401 rc1=0 and rc2=0 PASS 93 (8s) == sanityn test 94: signal vs CP callback race =========== 16:45:41 (1744145141) osc.lustre-OST0000-osc-ffff8800a9803000.idle_timeout=0 osc.lustre-OST0000-osc-ffff88012a6ad800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8800a9803000.idle_timeout=0 osc.lustre-OST0001-osc-ffff88012a6ad800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00887766 s, 923 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 2.99447 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5744: 498 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00422918 s, 969 kB/s osc.lustre-OST0000-osc-ffff8800a9803000.idle_timeout=debug osc.lustre-OST0000-osc-ffff88012a6ad800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800a9803000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a6ad800.idle_timeout=debug PASS 94 (10s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 16:45:51 (1744145151) llite.lustre-ffff8800a9803000.fast_read=0 llite.lustre-ffff88012a6ad800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0155324 s, 135 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8800a9803000.fast_read=1 llite.lustre-ffff88012a6ad800.fast_read=1 PASS 95a (18s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 16:46:09 (1744145169) llite.lustre-ffff8800a9803000.fast_read=0 llite.lustre-ffff88012a6ad800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB) copied, 0.0015551 s, 7.9 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8800a9803000.fast_read=1 llite.lustre-ffff88012a6ad800.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 16:46:17 (1744145177) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 16:46:18 (1744145178) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00148614 s, 2.8 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (2s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 16:46:20 (1744145180) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0214646 s, 97.7 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 16:46:22 (1744145182) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0184652 s, 114 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (1s) == sanityn test 100e: DoM: read on open and file size ==== 16:46:23 (1744145183) 1+0 records in 1+0 records out 12000 bytes (12 kB) copied, 0.00139798 s, 8.6 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 16:46:25 (1744145185) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.000450172 s, 9.1 MB/s PASS 101a (2s) == sanityn test 101b: Discard DoM data on rename ========= 16:46:27 (1744145187) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.000447863 s, 9.1 MB/s PASS 101b (1s) == sanityn test 101c: Discard DoM data on close-unlink === 16:46:28 (1744145188) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.000449287 s, 9.1 MB/s PASS 101c (3s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 16:46:31 (1744145191) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 00 02 00 00 00 b3 34 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000402:0x34b3:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.remote fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 40 02 00 00 00 51 47 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000402:0x4751:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.remote/subdir fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 00 02 00 00 00 b4 34 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000402:0x34b4:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.stripe fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 40 02 00 00 00 52 47 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000402:0x4752:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.stripe/subdir fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 40 02 00 00 00 53 47 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000402:0x4753:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! PASS 102 (3s) == sanityn test 103: Test size correctness with lockahead ========================================================== 16:46:34 (1744145194) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d103.sanityn fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1744145195 lockahead_test: lockahead_test.c:1191: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1744145197 Finishing test test23 at 1744145198 PASS 103 (7s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 16:46:41 (1744145201) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00224099 s, 457 kB/s oleg213-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg213-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps STAT a|m|ctime 1744145202 1744145202 1744145202 MDT a|m|ctime 1744145202 1744145202 1744145202 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.002806 s, 365 kB/s oleg213-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg213-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps STAT a|m|ctime 1744145202 1744145205 1744145205 MDT a|m|ctime 1744145202 1744145205 1744145205 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00836969 s, 122 kB/s oleg213-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg213-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps STAT a|m|ctime 1744145202 1744145209 1744145209 MDT a|m|ctime 1744145202 1744145209 1744145209 oleg213-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg213-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps oleg213-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg213-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps STAT a|m|ctime 1744145202 1744145209 1744145209 MDT a|m|ctime 1744145202 1744145209 1744145209 oleg213-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg213-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps STAT a|m|ctime 1744145202 1744145209 1744145218 MDT a|m|ctime 1744145202 1744145209 1744145218 PASS 104 (18s) == sanityn test 105: Glimpse and lock cancel race ======== 16:46:59 (1744145219) striped dir -i1 -c2 -H crush2 /mnt/lustre/d105.sanityn fail_loc=0x416 PASS 105 (23s) == sanityn test 106a: Verify the btime via statx() ======= 16:47:22 (1744145242) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 16:47:23 (1744145243) SKIP: sanityn test_106b statx() only test SKIP 106b (0s) == sanityn test 106c: Verify statx attributes mask ======= 16:47:23 (1744145243) SKIP: sanityn test_106c statx() only test SKIP 106c (1s) == sanityn test 107a: Basic grouplock conflict =========== 16:47:24 (1744145244) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0597914 s, 175 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6777 0x1a79 0x2c0000401 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7933 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7933 First grouplock blocks second one PASS 107a (4s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 16:47:28 (1744145248) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0651737 s, 161 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6789 0x1a85 0x280000401 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7933 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (9s) == sanityn test 108a: lseek: parallel updates ============ 16:47:37 (1744145257) 1+0 records in 1+0 records out 10000 bytes (10 kB) copied, 0.00211691 s, 4.7 MB/s fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB) copied, 4.14811 s, 2.0 MB/s PASS 108a (6s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 16:47:43 (1744145263) 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions oleg213-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 6 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions oleg213-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg213-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 8 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_size: failed to read coresize from /sys/module/acpi_cpufreq ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions oleg213-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 24 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions oleg213-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 29 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions oleg213-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg213-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 32 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions oleg213-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 34 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions oleg213-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg213-server: insmod: ERROR: could not insert module /home/green/git/lustre-release/lustre/../lnet/lnet/lnet.ko: File exists pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Iteration 35 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions oleg213-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 39 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions oleg213-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg213-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 45 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 192.168.202.113@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 PASS 109 (444s) == sanityn test 110: do not grant another lock on resend ========================================================== 16:55:07 (1744145707) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: '/mnt/lustre/f110.sanityn' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934275 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-08 16:55:08.000000000 -0400 Modify: 2025-04-08 16:55:08.000000000 -0400 Change: 2025-04-08 16:55:08.000000000 -0400 Birth: - fail_loc=0 fail_val=0 Connected clients: oleg213-client.virtnet oleg213-client.virtnet PASS 110 (65s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 16:56:12 (1744145772) fail_loc=0x8000018a /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6533: 4883 Terminated $MULTIOP $DIR2/$tdir/mdt0dir/foodir/file2 Ow4096_c PASS 111 (5s) == sanityn test 112: update max-inherit in default LMV === 16:56:17 (1744145777) lmv.lustre-clilmv-ffff880086ddb800.qos_threshold_rr=90 lmv.lustre-clilmv-ffff8800b6d1d000.qos_threshold_rr=90 PASS 112 (3s) == sanityn test 113: check servers of specified fs ======= 16:56:20 (1744145780) lustre-OST0000-osc-ffff880086ddb800 active. lustre-OST0001-osc-ffff880086ddb800 active. lustre-OST0000-osc-ffff880086ddb800 active. lustre-OST0001-osc-ffff880086ddb800 active. MGC192.168.202.113@tcp active. PASS 113 (1s) == sanityn test 114: implicit default LMV inherit ======== 16:56:21 (1744145781) /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 PASS 114 (6s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 16:56:27 (1744145787) total: 3000 create in 2.85 seconds: 1051.23 ops/second oleg213-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg213-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps oleg213-server: htree_dump_int_block: Ext2 directory block not found while calculating checksum for htree root oleg213-server: USE: f889 f1510 fail_loc=0x80002401 fail_val=5 mkdir: cannot create directory '/mnt/lustre/d115.sanityn/f1510': File exists PASS 115 (11s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 16:56:38 (1744145798) llite.lustre-ffff880086ddb800.intent_mkdir=1 llite.lustre-ffff8800b6d1d000.intent_mkdir=1 MD layout /mnt/lustre/d116.sanityn: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000bd0:0x9d8:0x0] 1 [0x240000404:0x92a:0x0] mkdir /mnt/lustre/d116.sanityn/tdir0 setdirstripe -D -i1 /mnt/lustre2/d116.sanityn/tdir0 mkdir /mnt/lustre/d116.sanityn/tdir0/tdir11 lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none mdtidx FID[seq:oid:ver] lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none mdtidx FID[seq:oid:ver] PASS 116 (2s) == sanityn test 200: service remains healthy while able to process request ========================================================== 16:56:40 (1744145800) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=14 striped dir -i0 -c2 -H crush2 /mnt/lustre/d200.sanityn fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.0079 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.0079 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.0063 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.0065 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.0039 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.0142 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.008 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.0135 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.1039 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 39.9968 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.0108 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.0112 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.0065 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.0083 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied1+0 records in 1+0 records out , 40.0221 s, 0.1 kB/s 4096 bytes (4.1 kB) copied, 40.0253 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.0159 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.1008 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.022 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.0258 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.0232 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.0297 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.0318 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.0225 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.0363 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.0357 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.1064 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 80.0307 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg213-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff880086ddb800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff880086ddb800.ost_server_uuid in FULL state after 0 sec PASS 200 (85s) cleanup: ====================================================== == sanityn test complete, duration 5375 sec ============== 16:58:05 (1744145885) === sanityn: start cleanup 16:58:06 (1744145886) === Stopping clients: oleg213-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg213-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 16:59:23 (1744145963) ===