-----============= acceptance-small: sanityn ============----- Thu Oct 23 23:11:33 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg128-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 23:11:41 (1761275501) === Starting client oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 Mount client oleg128-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 Started clients oleg128-client.virtnet: 192.168.201.128@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg128-client.virtnet: executing check_config_client /mnt/lustre oleg128-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg128-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff931f03c9b800.idle_timeout=debug osc.lustre-OST0000-osc-ffff931f20ea3000.idle_timeout=debug osc.lustre-OST0001-osc-ffff931f03c9b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff931f20ea3000.idle_timeout=debug disable quota as required oleg128-server: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanityn: finish setup 23:11:54 (1761275514) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214904 s, 48.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8189] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 23:11:55 (1761275515) PASS 0 (2s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 23:11:57 (1761275517) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (3s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 23:12:00 (1761275520) -rw-r--r-- 1 root root 0 Oct 23 23:12 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (2s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 23:12:02 (1761275522) -rw-r--r-- 1 root root 0 Oct 23 23:12 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (2s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 23:12:04 (1761275524) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (2s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 23:12:06 (1761275526) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (2s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 23:12:08 (1761275528) total 0 -rwxrwxrwx 1 root root 0 Oct 23 23:12 f2a -rwxrwxrwx 1 root root 0 Oct 23 23:12 f2b total 0 -rwxrwxrwx 1 root root 0 Oct 23 23:12 f2a -rwxrwxrwx 1 root root 0 Oct 23 23:12 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00316899 s, 162 kB/s PASS 2e (2s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 23:12:10 (1761275530) /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has type file OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has perms 0777 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by user #500 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by group #500 OK Can't lstat /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn: No such file or directory PASS 2f (2s) == sanityn test 2g: check blocks update on sync write ==== 23:12:12 (1761275532) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0506305 s, 41.4 MB/s /mnt/lustre/f2g.sanityn has 4096 blocks /mnt/lustre2/f2g.sanityn has 4096 blocks PASS 2g (2s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 23:12:14 (1761275534) PASS 3 (2s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 23:12:16 (1761275536) PASS 4 (3s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 23:12:19 (1761275539) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (2s) == sanityn test 6: remove of open file on other node ============================================================================ 23:12:21 (1761275541) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (1s) == sanityn test 7: remove of open directory on other node ======================================================================= 23:12:22 (1761275542) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (2s) == sanityn test 8: remove of open special file on other node ==================================================================== 23:12:24 (1761275544) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (2s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 23:12:26 (1761275546) PASS 9a (2s) == sanityn test 9b: append to striped sparse file ======== 23:12:28 (1761275548) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015441 s, 67.9 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0110983 s, 0.3 kB/s Data read (expecting 'foo'): foo PASS 9b (2s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 23:12:31 (1761275551) 1+0 records in 1+0 records out 1 byte copied, 0.00202113 s, 0.5 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00280248 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00283716 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00277465 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00289249 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00261515 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0026907 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0029646 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00259311 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00290595 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00278525 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00308324 s, 0.3 kB/s PASS 10a (3s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 23:12:33 (1761275553) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00356257 s, 862 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0122082 s, 336 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00035604 s, 8.6 MB/s PASS 10b (2s) == sanityn test 11: execution of file opened for write should return error ============================================================== 23:12:35 (1761275555) striped dir -i1 -c2 -H crush /mnt/lustre/d11 multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (2s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 23:12:37 (1761275557) warning: '-runtime' deprecated, use '-t runtime' instead - link 1849 (time 1761275568.49 total 10.00 last 184.84) - link 3508 (time 1761275578.49 total 20.01 last 165.86) - link 5370 (time 1761275588.50 total 30.01 last 186.11) - link 7067 (time 1761275598.50 total 40.02 last 169.60) - link 8809 (time 1761275608.50 total 50.02 last 174.16) - link 10000 (time 1761275614.48 total 56.00 last 199.20) - link 12166 (time 1761275624.48 total 66.00 last 216.56) - link 14256 (time 1761275634.49 total 76.00 last 208.92) - link 16498 (time 1761275644.49 total 86.00 last 224.18) - link 18669 (time 1761275654.49 total 96.01 last 217.04) total: 19532 link in 100.00 seconds: 195.31 ops/second using seed 2629932980 running for 100 seconds - stat 10000 (time 1761275564 ; total 5 ; last 5) - stat 20000 (time 1761275569 ; total 10 ; last 5) - stat 30000 (time 1761275574 ; total 15 ; last 5) - stat 40000 (time 1761275578 ; total 19 ; last 4) - stat 50000 (time 1761275583 ; total 24 ; last 5) - stat 60000 (time 1761275587 ; total 28 ; last 4) - stat 70000 (time 1761275592 ; total 33 ; last 5) - stat 80000 (time 1761275597 ; total 38 ; last 5) - stat 90000 (time 1761275601 ; total 42 ; last 4) - stat 100000 (time 1761275606 ; total 47 ; last 5) - stat 110000 (time 1761275614 ; total 55 ; last 8) - stat 117314 (time 1761275625 ; total 66 ; last 11) - stat 120000 (time 1761275628 ; total 69 ; last 3) - stat 127302 (time 1761275639 ; total 80 ; last 11) - stat 130000 (time 1761275642 ; total 83 ; last 3) - stat 136727 (time 1761275653 ; total 94 ; last 11) - stat 140000 (time 1761275657 ; total 98 ; last 4) total: 141129 stats in 100 seconds: 1411.290039 stats/second - unlinked 0 (time 1761275609 ; total 0 ; last 0) - unlinked 10000 (time 1761275666 ; total 57 ; last 57) unlink(/mnt/lustre2/lockdir/lockfile19532) error: No such file or directory total: 19531 unlinks in 82 seconds: 238.182922 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (22641) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (22643) - No such process PASS 12 (136s) == sanityn test 13: test directory page revocation ======= 23:14:53 (1761275693) striped dir -i1 -c2 -H crush /mnt/lustre/d13 f13 f13 PASS 13 (2s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 23:14:55 (1761275695) striped dir -i0 -c2 -H crush2 /mnt/lustre/d14aa.sanityn multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 23852 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (2s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 23:14:58 (1761275698) striped dir -i0 -c2 -H crush /mnt/lustre/d14ab.sanityn open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 24443 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (2s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 23:14:59 (1761275699) striped dir -i0 -c2 -H all_char /mnt/lustre/d14b.sanityn truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 25034 Terminated $DIR1/$tdir/sleep 60 PASS 14b (2s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 23:15:02 (1761275702) striped dir -i0 -c2 -H crush /mnt/lustre/d14c.sanityn cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 25627 Terminated $DIR1/$tdir/sleep 60 PASS 14c (3s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 23:15:04 (1761275704) striped dir -i0 -c2 -H all_char /mnt/lustre/d14d.sanityn chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 26221 Terminated $DIR1/$tdir/sleep 60 PASS 14d (2s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 23:15:06 (1761275706) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky Waiting for MDT destroys to complete 0 STRIPECOUNT=2 ORIGFREE=7207284 MAXFREE=2097152000 0 lustre-OST0000 avl=3684 grnt=5067 diff=-1383 limit=46331 FULL lustre-OST0001 avl=348 grnt=327 diff=21 limit=46331 FULL ERROR: dd blocks 7194150 != file size 7194172 LOG LOG2 file /tmp/oos2.log:dd: error writing '/mnt/lustre/oosfile': No space left on device /tmp/oos2.log:3598978+0 records in /tmp/oos2.log:3598977+0 records out /tmp/oos2.log:3685352448 bytes (3.7 GB, 3.4 GiB) copied, 482.042 s, 7.6 MB/s /tmp/oos2.log2:dd: error writing '/mnt/lustre2/oosfile2': No space left on device /tmp/oos2.log2:3595174+0 records in /tmp/oos2.log2:3595173+0 records out /tmp/oos2.log2:3681457152 bytes (3.7 GB, 3.4 GiB) copied, 482.396 s, 7.6 MB/s Waiting for MDT destroys to complete Waiting for MDT destroys to complete PASS 15 (518s) == sanityn test 16a: 2500 iterations of dual-mount fsx === 23:23:44 (1761276224) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0678753 s, 124 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0682536 s, 123 MB/s Chance of close/open is 1 in 50 Seed set to 6226 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fallocating to largest ever: 0x11e0ad fallocating to largest ever: 0x1f7494 truncating to largest ever: 0x46d091 fallocating to largest ever: 0x927841 truncating to largest ever: 0x9983a2 truncating to largest ever: 0x99de91 fallocating to largest ever: 0x9a6479 000100[0] 1761276227.480667 punch from 0x51b0f7 to 0x5212fc (0x6206 bytes) 000200[0] 1761276228.276474 mapread 0x15c538 thru 0x169059 (0xcb22 bytes) fallocating to largest ever: 0x9cb8d0 truncating to largest ever: 0x9e0d95 000300[1] 1761276229.117104 trunc from 0x301089 to 0x304c37 (0x3baf bytes) 000400[1] 1761276229.925433 fallocate 0x6c7e5c thru 0x6cd0e2 (0x5287 bytes) 000500[1] 1761276230.864847 read 0x7cde37 thru 0x7d528a (0x7454 bytes) 000600[1] 1761276231.786870 mapwrite 0x282570 thru 0x28868c (0x611d bytes) 000800[1] 1761276233.630162 read 0x6320a3 thru 0x632119 (0x77 bytes) 000900[1] 1761276234.457580 trunc from 0x9d5510 to 0x8aacf9 (0x12a816 bytes) 001000[1] 1761276235.357668 trunc from 0x4897b2 to 0x1e3310 (0x2a64a1 bytes) truncating to largest ever: 0x9ebdce fallocating to largest ever: 0x9ee05b 001300[0] 1761276238.260080 mapwrite 0x0d030a thru 0x0d1f3a (0x1c31 bytes) 001400[0] 1761276239.080113 mapwrite 0x7639fc thru 0x772cb6 (0xf2bb bytes) 001500[0] 1761276239.899093 fallocate 0x843676 thru 0x84a44a (0x6dd5 bytes) 001600[1] 1761276240.807967 fallocate 0x050e61 thru 0x053635 (0x27d5 bytes) 001700[1] 1761276241.599442 punch from 0x0a99f9 to 0x0aba66 (0x206e bytes) 001900[1] 1761276243.330445 trunc from 0x95b71f to 0x301bb9 (0x659b65 bytes) 002000[1] 1761276244.167410 mapread 0x14286c thru 0x14dfce (0xb763 bytes) fallocating to largest ever: 0x9f8458 002100[1] 1761276245.039952 mapread 0x18e5ae thru 0x1976b0 (0x9103 bytes) 002200[0] 1761276245.951473 read 0x1dabd4 thru 0x1e5194 (0xa5c1 bytes) 002400[0] 1761276247.597727 trunc from 0x66d0e3 to 0x6fe1fe (0x9111c bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 6248 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fallocating to largest ever: 0x5ecb64 fallocating to largest ever: 0x7e3819 truncating to largest ever: 0x951ce5 truncating to largest ever: 0x99d70e fallocating to largest ever: 0x9f1921 000100[0] 1761276249.357212 read_OD 0x361000 thru 0x364fff (0x4000 bytes) fallocating to largest ever: 0x9f7bf1 000400[0] 1761276251.374260 mapread 0x13f000 thru 0x143d03 (0x4d04 bytes) 000600[0] 1761276252.885172 trunc from 0x12c63d to 0x13211f (0x5ae3 bytes) 000800[0] 1761276254.452431 punch from 0x15a252 to 0x168937 (0xe6e6 bytes) 000900[0] 1761276255.239840 trunc from 0x23b34c to 0x247c7e (0xc933 bytes) fallocating to largest ever: 0xa00000 001200[0] 1761276257.137486 trunc from 0x0727d9 to 0x07d127 (0xa94f bytes) 001300[0] 1761276257.818252 trunc from 0x80388c to 0x0fc8d6 (0x706fb5 bytes) 001500[0] 1761276259.099943 punch from 0x115287 to 0x116a4f (0x17c9 bytes) 001600[1] 1761276259.922228 mapread 0x2c1000 thru 0x2cca8a (0xba8b bytes) 001800[1] 1761276261.209517 mapread 0x4ef000 thru 0x4f7a16 (0x8a17 bytes) 001900[1] 1761276261.995235 read_OD 0x0a6000 thru 0x0a6fff (0x1000 bytes) 002000[0] 1761276262.818622 punch from 0x60c908 to 0x61a38b (0xda84 bytes) 002100[0] 1761276263.287280 trunc from 0x7d1e4c to 0x4fcf94 (0x2d4eb7 bytes) 002200[0] 1761276263.929534 mapread 0x166000 thru 0x16e0c4 (0x80c5 bytes) 002300[1] 1761276264.693858 trunc from 0x7ef1c7 to 0x142ae0 (0x6ac6e6 bytes) 002400[1] 1761276265.365478 punch from 0x4f22bd to 0x4fa99a (0x86de bytes) 002500[1] 1761276266.117462 mapread 0x0f5000 thru 0x10090b (0xb90c bytes) All operations completed A-OK! PASS 16a (43s) == sanityn test 16b: 2500 iterations of dual-mount fsx at small size ========================================================== 23:24:27 (1761276267) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.068242 s, 123 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0608215 s, 138 MB/s Chance of close/open is 1 in 50 Seed set to 6268 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn skipping zero size read truncating to largest ever: 0x4e0b58 fallocating to largest ever: 0x9813fe 000200[1] 1761276270.566226 punch from 0x430f62 to 0x4327fe (0x189d bytes) 000300[1] 1761276271.156583 write 0x4d524e thru 0x4d600e (0xdc1 bytes) fallocating to largest ever: 0x98feff 000400[0] 1761276271.912926 trunc from 0x775246 to 0x3a210a (0x3d313b bytes) 000500[1] 1761276272.608513 trunc from 0x93a8a7 to 0x93c5aa (0x1d04 bytes) truncating to largest ever: 0x9f5f89 000600[0] 1761276273.412065 mapwrite 0x59be5a thru 0x59c063 (0x20a bytes) 000800[0] 1761276274.896149 mapwrite 0x17ebac thru 0x17fb98 (0xfed bytes) 000900[0] 1761276275.643611 fallocate 0x92dfca thru 0x92fb64 (0x1b9b bytes) 001000[0] 1761276276.463948 mapread 0x04b5be thru 0x04c22f (0xc72 bytes) 001100[1] 1761276277.247971 write 0x1d62d9 thru 0x1d6d95 (0xabd bytes) 001200[1] 1761276277.966367 trunc from 0x9a6611 to 0x2d036e (0x6d62a2 bytes) 001300[0] 1761276278.684479 punch from 0x1bae21 to 0x1bb0a2 (0x282 bytes) 001400[0] 1761276279.386592 read 0x3d598c thru 0x3d6b71 (0x11e6 bytes) 001500[1] 1761276280.054231 fallocate 0x9d8f07 thru 0x9da754 (0x184e bytes) 001800[0] 1761276282.165759 trunc from 0x4d3ee9 to 0x4d5756 (0x186e bytes) 001900[1] 1761276283.023195 write 0x090e46 thru 0x0912c2 (0x47d bytes) 002200[1] 1761276285.189409 trunc from 0x19ed07 to 0x19f465 (0x75f bytes) 002300[0] 1761276285.908928 mapread 0x34d8bf thru 0x34e3e0 (0xb22 bytes) 002400[1] 1761276286.556402 fallocate 0x81aa58 thru 0x81ac98 (0x241 bytes) truncating to largest ever: 0x9fc25c All operations completed A-OK! PASS 16b (22s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 23:24:49 (1761276289) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0633004 s, 133 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.063179 s, 133 MB/s Waiting for MDT destroys to complete Chance of close/open is 1 in 50 Seed set to 6300 fd 0: /mnt/lustre/f16c.sanityn fd 1: /mnt/lustre2/f16c.sanityn fallocating to largest ever: 0x272602 fallocating to largest ever: 0x368ecd fallocating to largest ever: 0x3b7621 truncating to largest ever: 0x9c42e9 000100[0] 1761276301.222689 write 0x7a5339 thru 0x7ad660 (0x8328 bytes) 000200[1] 1761276302.098329 trunc from 0x994e14 to 0x29f73d (0x6f56d6 bytes) 000300[0] 1761276303.061238 mapwrite 0x2f4a6a thru 0x2fbd83 (0x731a bytes) 000500[1] 1761276304.888585 mapread 0x8fac1c thru 0x904922 (0x9d07 bytes) 000600[1] 1761276305.731390 mapwrite 0x3fddd6 thru 0x404fe9 (0x7214 bytes) 000700[0] 1761276306.502747 fallocate 0x6ca803 thru 0x6cf7c8 (0x4fc6 bytes) 000800[0] 1761276307.313283 mapwrite 0x6abdcf thru 0x6b53ce (0x9600 bytes) 000900[1] 1761276308.105515 read 0x26a0c5 thru 0x2773f4 (0xd330 bytes) 001000[0] 1761276309.084257 fallocate 0x603bca thru 0x60c57a (0x89b1 bytes) 001100[1] 1761276309.962854 mapwrite 0x0fc47c thru 0x10356b (0x70f0 bytes) 001200[1] 1761276310.777254 write 0x60c131 thru 0x6171ea (0xb0ba bytes) truncating to largest ever: 0x9db179 truncating to largest ever: 0x9f2124 001300[0] 1761276311.677909 write 0x4ddd62 thru 0x4e5194 (0x7433 bytes) 001400[1] 1761276312.545360 mapwrite 0x52bebf thru 0x539d08 (0xde4a bytes) 001500[1] 1761276313.358097 trunc from 0x59bb1c to 0x49ff90 (0xfbb8b bytes) 001700[1] 1761276314.941496 write 0x92b12e thru 0x939f78 (0xee4b bytes) 001800[0] 1761276315.730441 trunc from 0x6d6fea to 0x1d62e6 (0x500d03 bytes) 001900[1] 1761276316.496689 punch from 0x8c81b5 to 0x8c89a4 (0x7f0 bytes) 002000[0] 1761276317.404729 read 0x1cd746 thru 0x1d07d8 (0x3093 bytes) 002100[0] 1761276318.154590 trunc from 0x526c01 to 0x11a071 (0x40cb8f bytes) fallocating to largest ever: 0xa00000 002200[1] 1761276318.983006 mapwrite 0x83e1b9 thru 0x84445b (0x62a3 bytes) 002300[1] 1761276319.797554 trunc from 0x60b51b to 0x900349 (0x2f4e2f bytes) 002400[1] 1761276320.741394 write 0x3d5bbe thru 0x3e3737 (0xdb7a bytes) 002500[0] 1761276321.682156 read 0x2b0ebe thru 0x2b5f86 (0x50c9 bytes) All operations completed A-OK! PASS 16c (35s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 23:25:24 (1761276324) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0608414 s, 138 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0574363 s, 146 MB/s ldlm.namespaces.MGC192.168.201.128@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff931f03c9b800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff931f20ea3000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff931f03c9b800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff931f20ea3000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff931f20ea3000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff931f20ea3000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.66577 s, 62.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.651708 s, 161 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.963449 s, 109 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.64539 s, 63.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.858798 s, 122 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.20215 s, 87.2 MB/s PASS 16d (10s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 23:25:34 (1761276334) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0956513 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124154 s, 84.5 MB/s PASS 16e (2s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 23:25:37 (1761276337) TIMEOUT PASS 16f (22s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 23:25:59 (1761276359) TIMEOUT PASS 16g (22s) == sanityn test 16h: mmap read after truncate file ======= 23:26:21 (1761276381) striped dir -i0 -c2 -H crush /mnt/lustre/d16h.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (2s) == sanityn test 16i: read after truncate file ============ 23:26:23 (1761276383) striped dir -i0 -c2 -H all_char /mnt/lustre/d16i.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (3s) == sanityn test 16j: race dio with buffered i/o ========== 23:26:26 (1761276386) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.309163 s, 84.8 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.366144 s, 71.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.405201 s, 64.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.197676 s, 133 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.270607 s, 96.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.239178 s, 110 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.347325 s, 75.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.410462 s, 63.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.456981 s, 57.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.823359 s, 31.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.840667 s, 31.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.241858 s, 108 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.268448 s, 97.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.345289 s, 75.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.374476 s, 70.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.29444 s, 89.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.321736 s, 81.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.408981 s, 64.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.418914 s, 62.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (7s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 23:26:33 (1761276393) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 skipping zero size read truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Thu Oct 23 23:26:40 EDT 2025 + fsx 2 done: rc=0 Thu Oct 23 23:26:40 EDT 2025 + fsx 3 done: rc=0 Thu Oct 23 23:26:40 EDT 2025 + fsx 4 done: rc=0 Thu Oct 23 23:26:40 EDT 2025 + fsx 5 done: rc=0 Thu Oct 23 23:26:40 EDT 2025 PASS 16k (10s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 23:26:43 (1761276403) fail_loc=0x8000030a PASS 17 (5s) == sanityn test 18: mmap sanity check =========================================================================================== 23:26:48 (1761276408) mmap test1: basic mmap operation (PASS, 0.028467s) mmap test2: MAP_PRIVATE not write back (PASS, 0.018217s) mmap test3: concurrent mmap ops on two nodes (PASS, 2.5807s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.0975s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.033797s) mmap test6: check mmap write/read content on two nodes (PASS, 0.037997s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.41635s) mmap test11: multi-thread shared mmap (PASS, 0.41977s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0198s) PASS 18 (11s) == sanityn test 19: test concurrent uncached read races ========================================================================= 23:26:59 (1761276419) 32+0 records in 32+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.185917 s, 90.2 MB/s loop 5 loop 10 loop 15 loop 20 PASS 19 (12s) == sanityn test 20: test extra readahead page left in cache ============================================================== 23:27:11 (1761276431) striped dir -i0 -c2 -H all_char /mnt/lustre/d20.sanityn PASS 20 (2s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 23:27:13 (1761276433) striped dir -i1 -c2 -H crush2 /mnt/lustre/d21.sanityn rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (3s) == sanityn test 23: others should see updated atime while another read============================================================== 23:27:16 (1761276436) now is 1761276436 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 new atime is 1761276497 PASS 23 (63s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 23:28:19 (1761276499) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2028 1269276 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 30732 3576288 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 8908 3598112 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 39640 7174400 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2028 1269276 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1784 1269520 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 30732 3576288 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 8908 3598112 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 39640 7174400 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 354 999.7K 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1000.0K 279 999.7K 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 256.0K 403 255.6K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 256.0K 370 255.6K 1% /mnt/lustre[OST:1] filesystem_summary: 511.9K 633 511.2K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 354 999.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1000.0K 279 999.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 256.0K 403 255.6K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 256.0K 370 255.6K 1% /mnt/lustre2[OST:1] filesystem_summary: 511.9K 633 511.2K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 1.3G 2.0M 1.2G 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1.3G 1.7M 1.2G 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3.7G 30.0M 3.4G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.7G 8.7M 3.4G 1% /mnt/lustre[OST:1] filesystem_summary: 7.3G 38.7M 6.8G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 354 1023646 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1024000 279 1023721 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 262144 403 261741 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 262144 370 261774 1% /mnt/lustre2[OST:1] filesystem_summary: 524148 633 523515 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2028 1269276 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 30732 3576288 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 8908 3598112 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 39640 7174400 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 354 999.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1000.0K 279 999.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 256.0K 403 255.6K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 256.0K 370 255.6K 1% /mnt/lustre2[OST:1] filesystem_summary: 511.9K 633 511.2K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 354 1023646 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1024000 279 1023721 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 262144 403 261741 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 262144 370 261774 1% /mnt/lustre[OST:1] filesystem_summary: 524148 633 523515 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 354 1023646 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1024000 279 1023721 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 262144 403 261741 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 262144 370 261774 1% /mnt/lustre2[OST:1] filesystem_summary: 524148 633 523515 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2028 1269276 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 30732 3576288 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 8908 3598112 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 39640 7174400 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2028 1269276 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1784 1269520 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 30732 3576288 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 8908 3598112 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 39640 7174400 1% /mnt/lustre2 PASS 24a (2s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 23:28:21 (1761276501) PASS 24b (2s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 23:28:23 (1761276503) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (3s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 23:28:26 (1761276506) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] PASS 25b (2s) == sanityn test 26a: allow mtime to get older ============ 23:28:28 (1761276508) utime: good mknod times 1761276507 <= 1761276508 <= 1761276508 for /mnt/lustre/f26a utime: good mknod times 1761276507 <= 1761276508 <= 1761276508 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (3s) == sanityn test 26b: sync mtime between ost and mds ====== 23:28:31 (1761276511) PASS 26b (4s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 23:28:35 (1761276515) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279613 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-10-23 23:28:36.000000000 -0400 Birth: 2025-10-23 23:28:36.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279613 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-10-23 23:28:36.000000000 -0400 Birth: 2025-10-23 23:28:36.000000000 -0400 PASS 26c (3s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 23:28:39 (1761276519) striped dir -i0 -c2 -H crush /mnt/lustre/d30.sanityn 0 PASS 30 (4s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 23:28:43 (1761276523) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d31a.sanityn fail_loc=0x314 PASS 31a (3s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 23:28:46 (1761276526) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg128-server mds-ost sync done. Waiting for MDT destroys to complete striped dir -i1 -c2 -H crush /mnt/lustre/d31b.sanityn fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.128@tcp:/lustre 7666232 39640 7174400 1% /mnt/lustre2 PASS 31b (10s) == sanityn test 31r: open-rename(replace) race =========== 23:28:56 (1761276536) -rw-r--r-- 1 root root 0 Oct 23 23:28 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (5s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 23:29:01 (1761276541) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 47780 Terminated tail -f $tfile PASS 31s (3s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 23:29:04 (1761276544) PASS 31t (3s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 23:29:07 (1761276547) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (1s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 23:29:09 (1761276549) Stopping /mnt/lustre-mds1 (opts:) on oleg128-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg128-server: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0000 create file on 2nd stripe under striped directory /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre/d33c.sanityn/striped touch /mnt/lustre2/d33c.sanityn/striped/subfile SoLC count 2 Below operations shouldn't trigger Solc: create file on 2nd stripe after setattr chmod 777 /mnt/lustre/d33c.sanityn/striped; touch /mnt/lustre2/d33c.sanityn/striped/subfile create remote dir and local dir /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote create local dir after remote dir creation transaction commit /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote.3; do_facet mds2 /home/green/git/lustre-release/lustre/utils/lctl set_param -n osd*.*MDT0001.force_sync 1; mkdir /mnt/lustre/d33c.sanityn/local.3 Solc count 0 PASS 33c (18s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 23:29:27 (1761276567) remote directory create /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre2/d33d.sanityn/subdir CoS count 2 cross-MDT rename mkdir /mnt/lustre/d33d.sanityn; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn/d1; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33d.sanityn/d2; touch /mnt/lustre/d33d.sanityn/d1/src mv /mnt/lustre2/d33d.sanityn/d1/src /mnt/lustre2/d33d.sanityn/d2/tgt CoS count 2 migrate /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs migrate -m 1 /mnt/lustre2/d33d.sanityn CoS count 2 PASS 33d (15s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 23:29:42 (1761276582) plain dir creation mkdir /mnt/lustre2/d33e.sanityn/plain open file and write echo abc > /mnt/lustre2/d33e.sanityn/f33e.sanityn append write echo dfg >> /mnt/lustre2/d33e.sanityn/f33e.sanityn setattr touch /mnt/lustre2/d33e.sanityn/f33e.sanityn file unlink rm /mnt/lustre2/d33e.sanityn/f33e.sanityn plain dir unlink rmdir /mnt/lustre2/d33e.sanityn/plain striped directory creation /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre2/d33e.sanityn/striped set default LMV to create striped subdir /home/green/git/lustre-release/lustre/utils/lfs setdirstripe -D -c 2 /mnt/lustre/d33e.sanityn striped subdir creation createmany -d /mnt/lustre/d33e.sanityn/subdir 100 total: 100 mkdir in 0.29 seconds: 339.44 ops/second sub file creation and write createmany -o /mnt/lustre/d33e.sanityn/subfile 100; echo abc > /mnt/lustre/d33e.sanityn/subfile1 total: 100 open/close in 0.28 seconds: 359.96 ops/second sub file append write echo dfg >> /mnt/lustre2/d33e.sanityn/subfile2 subdir setatttr touch /mnt/lustre2/d33e.sanityn/subdir1 subdir unlink unlinkmany -d /mnt/lustre/d33e.sanityn/subdir 100 - unlinked 0 (time 1761276583 ; total 0 ; last 0) total: 100 unlinks in 2 seconds: 50.000000 unlinks/second sub file unlink unlinkmany /mnt/lustre2/d33e.sanityn/subfile 100 - unlinked 0 (time 1761276585 ; total 0 ; last 0) total: 100 unlinks in 0 seconds: inf unlinks/second sub file creation follows striped dir chmod chmod 777 /mnt/lustre/d33e.sanityn/striped; touch /mnt/lustre/d33e.sanityn/striped/subfile striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn/striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn CoS count 0 PASS 33e (5s) == sanityn test 34: no lock timeout under IO ============= 23:29:47 (1761276587) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg128-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff931f03c9b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff931f03c9b800.ost_server_uuid in FULL state after 0 sec oleg128-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff931f03c9b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff931f03c9b800.ost_server_uuid in IDLE state after 0 sec oleg128-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff931f03c9b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff931f03c9b800.ost_server_uuid in IDLE state after 0 sec oleg128-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff931f03c9b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff931f03c9b800.ost_server_uuid in IDLE state after 0 sec oleg128-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff931f03c9b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff931f03c9b800.ost_server_uuid in IDLE state after 0 sec oleg128-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff931f03c9b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff931f03c9b800.ost_server_uuid in IDLE state after 0 sec PASS 34 (108s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 23:31:35 (1761276695) striped dir -i1 -c2 -H all_char /mnt/lustre/d35.sanityn fail_loc=0x80000317 Race attempt 0 Wait for 57767 57779 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1743: 57767 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.201.128@tcp:/lustre 7.4G 39M 6.9G 1% /mnt/lustre 192.168.201.128@tcp:/lustre 7.4G 39M 6.9G 1% /mnt/lustre2 - open/close 617 (time 1761276713.95 total 10.00 last 61.68) - open/close 1394 (time 1761276723.96 total 20.02 last 77.60) - open/close 1810 (time 1761276733.98 total 30.03 last 41.54) - open/close 2209 (time 1761276743.99 total 40.05 last 39.83) - open/close 2747 (time 1761276754.00 total 50.05 last 53.76) - open/close 3553 (time 1761276764.01 total 60.06 last 80.54) total: 4000 open/close in 67.98 seconds: 58.84 ops/second PASS 35 (76s) == sanityn test 36: handle ESTALE/open-unlink correctly == 23:32:52 (1761276772) striped dir -i0 -c2 -H crush /mnt/lustre/d36.sanityn Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg128-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.479447 s, 109 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(0) *** before(7174340) after_dd(7110780) after(7161980) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.430856 s, 122 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(1) *** before(7161980) after_dd(7114900) after(7166100) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.569006 s, 92.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(2) *** before(7166100) after_dd(7114900) after(7166100) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.661241 s, 79.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(3) *** before(7166100) after_dd(7114900) after(7166100) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.54262 s, 96.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(4) *** before(7166100) after_dd(7119020) after(7170220) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.589872 s, 88.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(5) *** before(7170220) after_dd(7114900) after(7166100) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.349128 s, 150 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(6) *** before(7166100) after_dd(7098420) after(7149620) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.526094 s, 99.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(7) *** before(7149620) after_dd(7114900) after(7166100) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.978842 s, 53.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(8) *** before(7166100) after_dd(7123140) after(7174340) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.465148 s, 113 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(9) *** before(7174340) after_dd(7110780) after(7161980) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.528101 s, 99.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(10) *** before(7161980) after_dd(7114900) after(7166100) PASS 36 (259s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 23:37:11 (1761277031) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d37.sanityn multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 - create 1465 (time 1761277045.03 total 10.00 last 146.43) - create 3025 (time 1761277055.03 total 20.01 last 156.00) - create 4981 (time 1761277065.03 total 30.01 last 195.53) - create 6947 (time 1761277075.03 total 40.01 last 196.55) - create 8908 (time 1761277085.04 total 50.02 last 195.96) total: 10000 create in 52.99 seconds: 188.70 ops/second PASS 37 (59s) == sanityn test 39a: file mtime does not change after rename ========================================================== 23:38:10 (1761277090) repeat after cancel_lru_locks PASS 39a (4s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 23:38:14 (1761277094) repeat after cancel_lru_locks PASS 39b (4s) == sanityn test 39c: check truncate mtime update ================================================================================ 23:38:18 (1761277098) repeat after cancel_lru_locks PASS 39c (4s) == sanityn test 39d: sync write should update mtime ====== 23:38:22 (1761277102) fail_loc=0x411 fail_loc=0 PASS 39d (3s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 23:38:25 (1761277105) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (10s) == sanityn test 40c: pdirops: link and others ======================================================================== 23:38:35 (1761277115) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (9s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 23:38:44 (1761277124) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (9s) == sanityn test 40e: pdirops: rename and others ======================================================================== 23:38:53 (1761277133) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (8s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 23:39:02 (1761277142) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (5s) == sanityn test 41b: pdirops: create vs create ======================================================================== 23:39:07 (1761277147) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 open: File exists Conflict PASS 41b (5s) == sanityn test 41c: pdirops: create vs link ======================================================================== 23:39:12 (1761277152) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (6s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 23:39:18 (1761277158) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 41d (5s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 23:39:23 (1761277163) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 41e (10s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 23:39:33 (1761277173) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41f (12s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 23:39:45 (1761277185) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41g (12s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 23:39:57 (1761277197) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41h (15s) == sanityn test 41i: reint_open: create vs create ======== 23:40:12 (1761277212) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (768s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 23:53:00 (1761277980) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (9s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 23:53:09 (1761277989) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 42b (9s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 23:53:18 (1761277998) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (9s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 23:53:27 (1761278007) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42d (9s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 23:53:36 (1761278016) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (16s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 23:53:53 (1761278033) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42f (28s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 23:54:20 (1761278060) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42g (25s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 23:54:45 (1761278085) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42h (16s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 23:55:01 (1761278101) PASS 43a (93s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 23:56:34 (1761278194) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43b (12s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 23:56:47 (1761278207) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43c (11s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 23:56:58 (1761278218) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (12s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 23:57:10 (1761278230) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43e (12s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 23:57:22 (1761278242) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (11s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 23:57:33 (1761278253) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (12s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 23:57:45 (1761278265) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43h (13s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 23:57:59 (1761278279) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43i (13s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 23:58:12 (1761278292) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (103s) == sanityn test 43k: unlink vs create ==================== 23:59:55 (1761278395) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (481s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 00:07:56 (1761278876) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Oct 24 00:08:00 EDT 2025 PASS 44a (6s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 00:08:02 (1761278882) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 open: File exists Conflict PASS 44b (6s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 00:08:08 (1761278888) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (6s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 00:08:14 (1761278894) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 Conflict PASS 44d (6s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 00:08:20 (1761278900) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 Conflict PASS 44e (6s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 00:08:26 (1761278906) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=8 Conflict PASS 44f (6s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 00:08:32 (1761278912) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=8 Conflict PASS 44g (6s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 00:08:38 (1761278918) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44h (6s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 00:08:44 (1761278924) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 lfs mkdir: dirstripe error on '/mnt/lustre2/f44i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f44i.sanityn': File exists Conflict PASS 44i (5s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 00:08:49 (1761278929) PASS 45a (32s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 00:09:21 (1761278961) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=28 Conflict PASS 45b (5s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 00:09:26 (1761278966) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 Conflict PASS 45c (6s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 00:09:32 (1761278972) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (5s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 00:09:37 (1761278977) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 Conflict PASS 45e (5s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 00:09:42 (1761278982) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=9 mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (6s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 00:09:48 (1761278988) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (5s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 00:09:53 (1761278993) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 Conflict PASS 45h (5s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 00:09:58 (1761278998) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 Conflict PASS 45i (5s) == sanityn test 45j: read vs rename ====================== 00:10:04 (1761279004) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (524s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 00:18:47 (1761279527) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (5s) == sanityn test 46b: pdirops: link vs create ======================================================================== 00:18:52 (1761279532) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 open: File exists Conflict PASS 46b (5s) == sanityn test 46c: pdirops: link vs link ======================================================================== 00:18:57 (1761279537) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (6s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 00:19:03 (1761279543) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 46d (5s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 00:19:08 (1761279548) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46e (7s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 00:19:15 (1761279555) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=8 Conflict PASS 46f (6s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 00:19:21 (1761279561) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=8 Conflict PASS 46g (6s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 00:19:27 (1761279567) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 Conflict PASS 46h (5s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 00:19:32 (1761279572) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 lfs mkdir: dirstripe error on '/mnt/lustre2/f46i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f46i.sanityn': File exists Conflict PASS 46i (5s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 00:19:37 (1761279577) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f47a.sanityn': File exists Conflict PASS 47a (5s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 00:19:42 (1761279582) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 open: File exists Conflict PASS 47b (6s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 00:19:48 (1761279588) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f47c.sanityn' to '/mnt/lustre2/f47c.sanityn-2': File exists Conflict PASS 47c (5s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 00:19:53 (1761279593) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 47d (5s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 00:19:58 (1761279598) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f47e.sanityn' with non-directory Conflict PASS 47e (4s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 00:20:03 (1761279603) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 47f (6s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 00:20:08 (1761279608) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 47g (6s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 00:20:14 (1761279614) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.00248534 s, 4.1 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (7s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 00:20:21 (1761279621) 0+1 records in 0+1 records out 158 bytes copied, 0.00407206 s, 38.8 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3510: kill: (284528) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 00:20:25 (1761279625) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00311293 s, 329 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101118 s, 104 MB/s 3145728 PASS 51b (15s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 00:20:40 (1761279640) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00551618 s, 186 kB/s PASS 51c (7s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 00:20:47 (1761279647) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105276 s, 99.6 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (3s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 00:20:50 (1761279650) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 1326 0x52e 0x2c0000401 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 1326 0x52e 0x2c0000401 read lease(1) has applied. PASS 51e (5s) == sanityn test 54: rename locking ======================= 00:20:55 (1761279655) llite.lustre-ffff931f03c9b800.xattr_cache=1 llite.lustre-ffff931f20ea3000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762681 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 00:20:55.000000000 -0400 Modify: 2025-10-24 00:20:55.000000000 -0400 Change: 2025-10-24 00:20:55.000000000 -0400 Birth: 2025-10-24 00:20:55.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281272 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 00:21:01.000000000 -0400 Modify: 2025-10-24 00:21:01.000000000 -0400 Change: 2025-10-24 00:21:01.000000000 -0400 Birth: 2025-10-24 00:21:01.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762684 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 00:21:07.000000000 -0400 Modify: 2025-10-24 00:21:07.000000000 -0400 Change: 2025-10-24 00:21:07.000000000 -0400 Birth: 2025-10-24 00:21:07.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281273 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 00:21:07.000000000 -0400 Modify: 2025-10-24 00:21:07.000000000 -0400 Change: 2025-10-24 00:21:07.000000000 -0400 Birth: 2025-10-24 00:21:07.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762685 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 00:21:12.000000000 -0400 Modify: 2025-10-24 00:21:13.000000000 -0400 Change: 2025-10-24 00:21:13.000000000 -0400 Birth: 2025-10-24 00:21:12.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281275 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 00:21:12.000000000 -0400 Modify: 2025-10-24 00:21:13.000000000 -0400 Change: 2025-10-24 00:21:13.000000000 -0400 Birth: 2025-10-24 00:21:12.000000000 -0400 PASS 54 (25s) == sanityn test 55a: rename vs unlink target dir ========= 00:21:20 (1761279680) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (8s) == sanityn test 55b: rename vs unlink source dir ========= 00:21:28 (1761279688) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 00:21:36 (1761279696) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 00:21:49 (1761279709) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (11s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 00:22:00 (1761279720) fail_loc=0x155 We survived after AB/BA race test fail_loc=0x0 PASS 55e (13s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 00:22:13 (1761279733) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (14s) == sanityn test 55g: rename: race with trylock =========== 00:22:27 (1761279747) mdt.lustre-MDT0000.enable_rename_trylock=1 mdt.lustre-MDT0001.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (15s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 00:22:43 (1761279763) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1761279763 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 20.3464 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (5s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 00:22:48 (1761279768) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1761279770 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 6.10371 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (25s) == sanityn test 60: Verify data_version behaviour ======== 00:23:13 (1761279793) striped dir -i0 -c2 -H crush /mnt/lustre/d60.sanityn PASS 60 (3s) == sanityn test 70a: cd directory && rm directory ======== 00:23:16 (1761279796) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 00:23:19 (1761279799) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 00:23:22 (1761279802) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00253231 s, 25.9 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00244198 s, 26.8 MB/s File: /mnt/lustre2/f71a.sanityn Size: 262144 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205289281315 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 00:23:22.000000000 -0400 Modify: 2025-10-24 00:23:22.000000000 -0400 Change: 2025-10-24 00:23:22.000000000 -0400 Birth: 2025-10-24 00:23:22.000000000 -0400 46246 extent 0 in offset 65536, length 65536 flags (0x80000000): NET extent 1 in offset 196608, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 2, sum of lengths 131072, expected sum 131072 46247 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00411666 s, 15.9 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00386752 s, 16.9 MB/s File: /mnt/lustre/f71a.sanityn Size: 262144 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205289281315 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 00:23:22.000000000 -0400 Modify: 2025-10-24 00:23:22.000000000 -0400 Change: 2025-10-24 00:23:22.000000000 -0400 Birth: 2025-10-24 00:23:22.000000000 -0400 46248 extent 0 in offset 65536, length 65536 flags (0x80000000): NET extent 1 in offset 196608, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 2, sum of lengths 131072, expected sum 131072 46247 PASS 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 00:23:25 (1761279805) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00472986 s, 13.9 MB/s extent 0 in offset 0, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 1, sum of lengths 65536, expected sum 65536 PASS 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 00:23:28 (1761279808) filefrag 1.47.1-wc2 (08-Nov-2024) check last flag for file with 291 extents 290: 296960.. 296963: 370688.. 370691: 4: 0000: last,net,eof check last flag for file with 292 extents 291: 297984.. 297987: 445440.. 445443: 4: 0001: last,net,eof check last flag for file with 293 extents 292: 299008.. 299011: 569344.. 569347: 4: 0000: last,net,eof PASS 71c (16s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 00:23:45 (1761279825) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00627447 s, 10.4 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00112003 s, 58.5 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00116506 s, 56.3 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00976132 s, 6.7 MB/s disk usage: 262144 /mnt/lustre/f71d.sanityn file size: 458752 /mnt/lustre/f71d.sanityn extent 0 in offset 0, length 65536 flags (0x80000000): NET extent 1 in offset 131072, length 65536 flags (0x80000000): NET extent 2 in offset 262144, length 65536 flags (0x80000000): NET extent 3 in offset 393216, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 4, sum of lengths 262144, expected sum 262144 PASS 71d (14s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 00:23:58 (1761279838) llite.lustre-ffff931f03c9b800.xattr_cache=1 llite.lustre-ffff931f20ea3000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (4s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 00:24:02 (1761279842) llite.lustre-ffff931f03c9b800.xattr_cache=1 llite.lustre-ffff931f20ea3000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (4s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 00:24:06 (1761279846) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00480662 s, 213 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00699848 s, 146 kB/s 300024: get lock2 300026: get lock1 300024: done 300024: get lock1 300026: done 300026: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 300026: cannot get lock2: Resource deadlock avoided 300026: exit rc=0 300024: done 300024: exit rc=0 PASS 74 (7s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 00:24:13 (1761279853) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.00950343 s, 221 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00753796 s, 139 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4193: 300615 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (8s) == sanityn test 76: Verify MDT open_files listing ======== 00:24:21 (1761279861) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d76.sanityn opening files: ................ closing files: ................ PASS 76 (42s) == sanityn test 77a: check FIFO NRS policy =============== 00:25:03 (1761279903) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.084086 s, 200 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023816 s, 44.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339032 s, 30.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239376 s, 43.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310272 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429714 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489883 s, 21.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456938 s, 22.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513854 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041563 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049965 s, 21.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521408 s, 20.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355446 s, 29.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184676 s, 56.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230861 s, 45.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469442 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430634 s, 24.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195609 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169769 s, 61.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190277 s, 55.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00603376 s, 174 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192261 s, 54.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170793 s, 61.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00821336 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392939 s, 26.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156899 s, 66.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101236 s, 104 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187159 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00723247 s, 145 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335789 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015927 s, 65.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469035 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019033 s, 55.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=1 removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg128-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg128-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (4s) == sanityn test 77b: check CRR-N NRS policy ============== 00:25:07 (1761279907) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0822674 s, 204 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322249 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386937 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468385 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223044 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378573 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473037 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518305 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255109 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901806 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537561 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866484 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946567 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633593 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078848 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777898 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105511 s, 9.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214979 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207227 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248972 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322858 s, 32.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396775 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259208 s, 40.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244245 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385687 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042626 s, 24.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348041 s, 30.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00577457 s, 182 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371906 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376685 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345251 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388256 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405778 s, 25.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg128-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg128-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0919634 s, 182 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229499 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279239 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427601 s, 24.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474493 s, 22.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422478 s, 24.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287844 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396076 s, 26.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593294 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585024 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654776 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552065 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441342 s, 23.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526418 s, 19.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361871 s, 29.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383295 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460514 s, 22.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213791 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250216 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181219 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258182 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026233 s, 40.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258548 s, 40.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222598 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334265 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375624 s, 27.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339653 s, 30.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124411 s, 84.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149079 s, 70.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00906745 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00875951 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125273 s, 83.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143066 s, 73.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=3 removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg128-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg128-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (5s) == sanityn test 77c: check ORR NRS policy ================ 00:25:12 (1761279912) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.147913 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170248 s, 61.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306223 s, 34.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295996 s, 35.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379373 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034528 s, 30.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406282 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329036 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327796 s, 32.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245832 s, 42.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305401 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339877 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493697 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448995 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332981 s, 31.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455235 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436816 s, 24.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126579 s, 82.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142072 s, 73.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209243 s, 50.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130293 s, 80.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257691 s, 40.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265598 s, 39.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119101 s, 88.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268155 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0039703 s, 264 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310905 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297216 s, 35.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108364 s, 96.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00956201 s, 110 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248726 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140462 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235936 s, 44.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg128-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg128-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0908295 s, 185 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430319 s, 24.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495666 s, 21.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548231 s, 19.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399192 s, 26.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579244 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446922 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598849 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503874 s, 20.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527495 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070112 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581098 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063097 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593471 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595452 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669747 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593398 s, 17.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239448 s, 43.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317686 s, 33.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219831 s, 47.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033845 s, 31.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311253 s, 33.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282482 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300356 s, 34.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318066 s, 33.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322504 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04085 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254548 s, 41.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263563 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163577 s, 64.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321594 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142554 s, 73.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140353 s, 74.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=1 removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg128-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg128-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0674603 s, 249 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186693 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357939 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335635 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428096 s, 24.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534409 s, 19.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474082 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465025 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525845 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560137 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013639 s, 76.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291829 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360622 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319131 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588043 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369478 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598893 s, 17.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018158 s, 57.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174351 s, 60.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196176 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197667 s, 53.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168409 s, 62.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00356163 s, 294 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022757 s, 46.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316356 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153435 s, 68.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200224 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00656004 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149622 s, 70.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013858 s, 75.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173748 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00748142 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122007 s, 85.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg128-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg128-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (6s) == sanityn test 77d: check TRR nrs policy ================ 00:25:18 (1761279918) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.181975 s, 92.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149304 s, 70.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246295 s, 42.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370259 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342125 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322608 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218031 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295481 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449833 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747209 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581387 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531141 s, 19.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752989 s, 13.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689642 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069574 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595085 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862852 s, 12.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260673 s, 40.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295795 s, 35.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033493 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299147 s, 35.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220801 s, 47.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037307 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291609 s, 36.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325573 s, 32.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00818391 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00403316 s, 260 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128748 s, 81.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451737 s, 23.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366117 s, 28.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423875 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00425773 s, 246 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00420772 s, 249 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=1 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg128-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg128-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0870357 s, 193 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167556 s, 62.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317811 s, 33.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035526 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033484 s, 31.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336172 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441135 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507775 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309945 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428248 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033259 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347157 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272057 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473245 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216098 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383883 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469496 s, 22.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133388 s, 78.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224507 s, 46.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207743 s, 50.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203613 s, 51.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206786 s, 50.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140314 s, 74.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278937 s, 37.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00994657 s, 105 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218292 s, 48.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266692 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00393029 s, 267 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198392 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00618877 s, 169 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254805 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00504671 s, 208 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198264 s, 52.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg128-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg128-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0810316 s, 207 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125958 s, 83.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375571 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343154 s, 30.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522823 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565071 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479386 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035394 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431449 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449535 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617737 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346218 s, 30.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568296 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03583 s, 29.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705511 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573857 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487529 s, 21.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346119 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0028832 s, 364 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295514 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427521 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470479 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0031563 s, 332 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469234 s, 22.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344983 s, 30.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403884 s, 26.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383336 s, 27.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192009 s, 54.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532087 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122013 s, 85.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555993 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501575 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543399 s, 19.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=8 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg128-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg128-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (6s) == sanityn test 77e: check TBF NID nrs policy ============ 00:25:24 (1761279924) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.201.28@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.178739 s, 93.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276995 s, 37.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383028 s, 27.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383547 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454202 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338414 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432927 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044109 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419997 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424041 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568905 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056802 s, 18.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065271 s, 16.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842655 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743255 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483009 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573927 s, 18.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223811 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239033 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278675 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226123 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262338 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253295 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275371 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187053 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032184 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160635 s, 65.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00606762 s, 173 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0078079 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370186 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00545055 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174292 s, 60.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00583606 s, 180 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg128-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg128-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.104251 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161633 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311403 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334893 s, 31.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130181 s, 80.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345767 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333315 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042005 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295153 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322599 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00748502 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336473 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181525 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503115 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061721 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470659 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653198 s, 16.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138729 s, 75.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00478008 s, 219 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260469 s, 40.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017851 s, 58.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157387 s, 66.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166652 s, 62.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137162 s, 76.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207677 s, 50.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241153 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321822 s, 32.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223007 s, 47.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360594 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368189 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296006 s, 35.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318187 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340511 s, 30.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=2 removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg128-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg128-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.087659 s, 191 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015294 s, 68.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287588 s, 36.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379413 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033681 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314705 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306942 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371495 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500582 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497261 s, 21.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01331 s, 78.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505964 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352697 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420955 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315441 s, 33.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622706 s, 16.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763066 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00441906 s, 237 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00823193 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131321 s, 79.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101121 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121951 s, 86.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146178 s, 71.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327378 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316451 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00455993 s, 230 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354292 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00506859 s, 207 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023556 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00528798 s, 198 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214811 s, 48.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216178 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277994 s, 37.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=4 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg128-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg128-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0937167 s, 179 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163613 s, 64.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301763 s, 34.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252329 s, 41.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406142 s, 25.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173924 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372513 s, 28.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390275 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333639 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218592 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420527 s, 24.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535416 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311345 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629363 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694481 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521283 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438124 s, 23.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020278 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201257 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181274 s, 57.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186366 s, 56.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182603 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00454161 s, 231 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257651 s, 40.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022294 s, 47.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233634 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313565 s, 33.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370884 s, 28.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303998 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301169 s, 34.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00456152 s, 230 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318563 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00628372 s, 167 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=2 removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg128-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg128-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (10s) == sanityn test 77f: check TBF JobID nrs policy ========== 00:25:34 (1761279934) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg128-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.129493 s, 130 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226657 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129785 s, 80.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345114 s, 30.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548257 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531994 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475657 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577166 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626043 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548797 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654584 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590111 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653076 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895256 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841238 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570945 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759778 s, 13.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203884 s, 51.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211591 s, 49.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297702 s, 35.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113277 s, 92.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00592082 s, 177 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293263 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327168 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310811 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287564 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00575304 s, 182 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163694 s, 64.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246453 s, 42.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217372 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128132 s, 81.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00599487 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250139 s, 41.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg128-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg128-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg128-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0704536 s, 238 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015545 s, 67.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201064 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023494 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284848 s, 36.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034452 s, 30.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360194 s, 29.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368493 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284904 s, 36.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040989 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516342 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458905 s, 22.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423115 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322446 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245748 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220467 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259328 s, 40.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185701 s, 56.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151167 s, 69.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023363 s, 44.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167646 s, 62.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240017 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264236 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216221 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206172 s, 50.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202678 s, 51.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207407 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126277 s, 83.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00778555 s, 135 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00552105 s, 190 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00358808 s, 292 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00757831 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00337925 s, 310 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg128-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg128-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg128-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0599087 s, 280 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187356 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301241 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276103 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355977 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424126 s, 24.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026979 s, 38.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328069 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395299 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414936 s, 25.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489945 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489513 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548673 s, 19.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621809 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465139 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577134 s, 18.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728386 s, 14.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157731 s, 66.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119662 s, 87.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012072 s, 86.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00954059 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131787 s, 79.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194694 s, 53.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213189 s, 49.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171034 s, 61.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00948117 s, 111 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242631 s, 43.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00974466 s, 108 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00850109 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114407 s, 91.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00400333 s, 262 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00438116 s, 239 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00319562 s, 328 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg128-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg128-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg128-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0889458 s, 189 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220782 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409817 s, 25.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445744 s, 23.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496491 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04247 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290605 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234206 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403733 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169389 s, 61.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668478 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582399 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659963 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663427 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777152 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881843 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576487 s, 18.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220055 s, 47.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290915 s, 36.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342067 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035707 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869313 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428812 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509304 s, 20.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048126 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100509 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515333 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419611 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446321 s, 23.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444641 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518291 s, 20.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162216 s, 64.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934465 s, 11.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg128-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg128-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (9s) == sanityn test 77g: Change TBF type directly ============ 00:25:43 (1761279943) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (5s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 00:25:48 (1761279948) oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg128-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg128-client: oleg128-server: ssh exited with exit code 2 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg128-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg128-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg128-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 PASS 77h (5s) == sanityn test 77i: Change rank of TBF rule ============= 00:25:53 (1761279953) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (9s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 00:26:02 (1761279962) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0820106 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024386 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236143 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371478 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289052 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427545 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316871 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453234 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464956 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307472 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425707 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405261 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040902 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207964 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398467 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658435 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382728 s, 27.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754629 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071847 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782234 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791131 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766922 s, 13.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845366 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749583 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696213 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072942 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808295 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758994 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923459 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778294 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834094 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808898 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859821 s, 12.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=3 removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg128-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg128-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92791 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.443 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (44s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 00:26:46 (1761280006) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg128-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg128-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg128-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg128-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg128-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0805461 s, 208 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133495 s, 78.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172226 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023596 s, 44.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325674 s, 32.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342735 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343206 s, 30.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224713 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354419 s, 29.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361235 s, 29.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028807 s, 36.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227186 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365827 s, 28.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303367 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364537 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277281 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048933 s, 21.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148149 s, 70.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012955 s, 80.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00773327 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182761 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216798 s, 48.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022327 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162026 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228138 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253232 s, 41.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00679769 s, 154 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00816268 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254361 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021816 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243912 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00290887 s, 360 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00384028 s, 273 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg128-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg128-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6536 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9673 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg128-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg128-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg128-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg128-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg128-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0683699 s, 245 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169983 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263503 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350731 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331579 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401568 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211236 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232388 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209061 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421092 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561846 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225761 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548423 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259456 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437889 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595908 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521017 s, 20.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257344 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288254 s, 36.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289986 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284344 s, 36.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235133 s, 44.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300361 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220047 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261857 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373273 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335232 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185173 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037796 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110839 s, 94.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172231 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00539716 s, 194 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173994 s, 60.3 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg128-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg128-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6436 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9686 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (118s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 00:28:44 (1761280124) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg128-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg128-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg128-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg128-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg128-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0507551 s, 331 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012613 s, 83.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164357 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205665 s, 51.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250762 s, 41.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314902 s, 33.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238337 s, 44.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033685 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373963 s, 28.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365421 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195801 s, 53.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198441 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253848 s, 41.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296274 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303686 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203525 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474412 s, 22.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012536 s, 83.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179467 s, 58.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226949 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233219 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162932 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245257 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217052 s, 48.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123047 s, 85.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240911 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205196 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0044521 s, 236 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259871 s, 40.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269393 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256324 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00405228 s, 259 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00621778 s, 169 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg128-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg128-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6531 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9616 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg128-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg128-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg128-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg128-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg128-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0800263 s, 210 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370805 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213418 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394344 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338448 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466932 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449735 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284643 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356922 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453862 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517771 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568069 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495897 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680312 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512263 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056532 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688473 s, 15.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170691 s, 61.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242104 s, 43.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255633 s, 41.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02804 s, 37.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182547 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353426 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365362 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257462 s, 40.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00818695 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00632971 s, 166 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00721738 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117591 s, 89.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169367 s, 61.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115403 s, 90.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015143 s, 69.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00541537 s, 194 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg128-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg128-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6526 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9369 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (118s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 00:30:42 (1761280242) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg128-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0898887 s, 187 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252607 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276131 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242412 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529755 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046676 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442451 s, 23.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345991 s, 30.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537165 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134088 s, 78.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475695 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215436 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492417 s, 21.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656513 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402449 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806045 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506699 s, 20.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185246 s, 56.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017441 s, 60.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188435 s, 55.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194899 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219852 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181634 s, 57.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245308 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00453565 s, 231 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265655 s, 39.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0080297 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264234 s, 39.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113766 s, 92.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00575985 s, 182 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390023 s, 26.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151333 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130767 s, 80.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg128-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg128-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90453 s, 21.4 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73547 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.28@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.28@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0568542 s, 295 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135292 s, 77.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00859759 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369173 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037525 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316098 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407015 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380211 s, 27.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122332 s, 85.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348588 s, 30.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313629 s, 33.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456187 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429701 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022534 s, 46.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329846 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430961 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168888 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102055 s, 103 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201612 s, 52.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214391 s, 48.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258385 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270268 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178105 s, 58.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241508 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277339 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00382222 s, 274 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011764 s, 89.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206055 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242552 s, 43.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251767 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00413483 s, 254 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00482715 s, 217 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307238 s, 34.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg128-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg128-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90168 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73494 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.28@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg128-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0620976 s, 270 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00931337 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309911 s, 33.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313819 s, 33.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332413 s, 31.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046461 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442549 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559734 s, 18.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505412 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331431 s, 31.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406415 s, 25.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417965 s, 25.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041217 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274207 s, 38.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499754 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661495 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489531 s, 21.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186452 s, 56.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218919 s, 47.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167841 s, 62.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229349 s, 45.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273028 s, 38.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204465 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022351 s, 46.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116796 s, 89.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303744 s, 34.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00771814 s, 136 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00482431 s, 217 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00872542 s, 120 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014792 s, 70.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173479 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120219 s, 87.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00561248 s, 187 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg128-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg128-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90544 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.88157 s, 21.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg128-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0617801 s, 272 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172919 s, 60.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276319 s, 37.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400873 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433827 s, 24.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373889 s, 28.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318501 s, 32.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048086 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339967 s, 30.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440938 s, 23.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440221 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424722 s, 24.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387407 s, 27.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380484 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268576 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327388 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477914 s, 21.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173611 s, 60.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213892 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165377 s, 63.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218885 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273211 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00599978 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253078 s, 41.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202411 s, 51.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265734 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161787 s, 64.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100188 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177986 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00460579 s, 228 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320013 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167571 s, 62.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00190166 s, 551 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg128-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg128-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75662 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73224 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.88795 s, 21.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73326 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg128-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0653648 s, 257 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275393 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256839 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345622 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356115 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053196 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325482 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171803 s, 61.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546195 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601299 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489198 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595374 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568799 s, 18.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414194 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463396 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622803 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583278 s, 18.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145078 s, 72.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160318 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182105 s, 57.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281241 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194063 s, 54.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279647 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353182 s, 29.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168574 s, 62.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147214 s, 71.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265243 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142318 s, 73.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214106 s, 49.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200995 s, 52.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179128 s, 58.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0093373 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320236 s, 32.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg128-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg128-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4557 s, 5.4 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4375 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg128-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0664088 s, 253 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174787 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192435 s, 54.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298978 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281131 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02219 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475667 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301429 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038675 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328588 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518775 s, 20.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627856 s, 16.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611008 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849075 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835506 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567224 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531145 s, 19.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278739 s, 37.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222753 s, 47.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017021 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265752 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324328 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291809 s, 35.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238775 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265837 s, 39.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364732 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041404 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414603 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309498 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398707 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351453 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00300103 s, 349 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00312964 s, 335 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=1 removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg128-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg128-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90666 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73356 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg128-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0639132 s, 263 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030704 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382547 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039713 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373292 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301333 s, 34.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437086 s, 24.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382831 s, 27.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032344 s, 32.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527412 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516492 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525117 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325971 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519666 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723846 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647175 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038194 s, 27.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190938 s, 54.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019977 s, 52.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166739 s, 62.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02444 s, 42.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00977979 s, 107 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283121 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169106 s, 62.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239664 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150677 s, 69.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00731907 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136572 s, 76.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351788 s, 29.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330924 s, 31.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00715837 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00870527 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151039 s, 69.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg128-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg128-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75302 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7312 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75044 s, 10.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74212 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (271s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 00:35:13 (1761280513) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg128-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0624888 s, 268 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117674 s, 89.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435689 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357387 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402999 s, 26.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335285 s, 31.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305949 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381274 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237781 s, 44.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603802 s, 17.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566286 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315768 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030749 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331017 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387823 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675585 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060598 s, 17.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190441 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204184 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252694 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216381 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264139 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271035 s, 38.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216063 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00945438 s, 111 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01487 s, 70.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432107 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208477 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163698 s, 64.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451155 s, 23.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188875 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316504 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399706 s, 26.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg128-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg128-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90123 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73433 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn Waiting for MDT destroys to complete PASS 77kb (26s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 00:35:39 (1761280539) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.28@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.28@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0582302 s, 288 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00943287 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163418 s, 64.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254766 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297649 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289038 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298998 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463252 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248291 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032534 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018124 s, 57.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481693 s, 21.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338264 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466021 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564453 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454294 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422911 s, 24.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118927 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125197 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109868 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120263 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117623 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120948 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128328 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127864 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129038 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122524 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120023 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127775 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120119 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120728 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128676 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127412 s, 8.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg128-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg128-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90877 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74245 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn Waiting for MDT destroys to complete PASS 77kc (32s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 00:36:11 (1761280571) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.28@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg128-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0700923 s, 239 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030928 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380435 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309119 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283856 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282549 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153325 s, 68.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044268 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045434 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540561 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484259 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036483 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448741 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840627 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821171 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650909 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783224 s, 13.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=3 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170905 s, 61.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273525 s, 38.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231862 s, 45.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337038 s, 31.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308797 s, 34.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347666 s, 30.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298243 s, 35.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216905 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353934 s, 29.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282836 s, 37.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157564 s, 66.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170704 s, 61.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00543341 s, 193 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447315 s, 23.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103671 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039763 s, 26.4 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg128-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg128-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90862 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97182 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn Waiting for MDT destroys to complete PASS 77kd (26s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 00:36:37 (1761280597) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg128-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.066017 s, 254 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176597 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392857 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427248 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309765 s, 33.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384067 s, 27.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429372 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411518 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495764 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558786 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717013 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734794 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632477 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620732 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590879 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725414 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723403 s, 14.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130871 s, 80.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195551 s, 53.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136377 s, 76.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213816 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322634 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109638 s, 95.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263632 s, 39.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315757 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300248 s, 34.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00577611 s, 182 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355158 s, 29.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00561157 s, 187 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122445 s, 85.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131515 s, 79.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015307 s, 68.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00784648 s, 134 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg128-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg128-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75858 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74226 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90343 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.745 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn Waiting for MDT destroys to complete PASS 77ke (60s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 00:37:37 (1761280657) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg128-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0610013 s, 275 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115286 s, 91.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290229 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297711 s, 35.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256578 s, 40.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342576 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204747 s, 51.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335432 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032117 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428715 s, 24.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404161 s, 25.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253721 s, 41.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219109 s, 47.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227302 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200213 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195611 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185092 s, 56.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167236 s, 62.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223493 s, 46.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285191 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278876 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236397 s, 44.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210396 s, 49.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015889 s, 66.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286938 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178875 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00363977 s, 288 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107177 s, 97.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327735 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119046 s, 88.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00673746 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161401 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00663584 s, 158 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg128-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg128-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6475 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9606 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn Waiting for MDT destroys to complete PASS 77kf (51s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 00:38:28 (1761280708) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg128-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0643466 s, 261 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135578 s, 77.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376698 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392164 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122534 s, 85.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271444 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251689 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374805 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427373 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332019 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234458 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03938 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512462 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506102 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548232 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679894 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633019 s, 16.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00466621 s, 225 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156164 s, 67.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213711 s, 49.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204215 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183995 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258354 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00366709 s, 286 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147699 s, 71.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00388604 s, 270 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119764 s, 87.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00739809 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025609 s, 40.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149708 s, 70.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260992 s, 40.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00814712 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159071 s, 65.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=1 removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg128-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg128-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90425 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73693 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg128-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0644087 s, 260 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223288 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284316 s, 36.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312864 s, 33.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446817 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414703 s, 25.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384682 s, 27.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364038 s, 28.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469993 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360595 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036548 s, 28.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391121 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259261 s, 40.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478173 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389583 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461503 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630707 s, 16.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198264 s, 52.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258497 s, 40.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210148 s, 49.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349896 s, 30.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314604 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300119 s, 34.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374898 s, 28.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132144 s, 79.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037917 s, 27.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266252 s, 39.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103814 s, 101 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00618454 s, 170 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394655 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130993 s, 80.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102124 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00833168 s, 126 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg128-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg128-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76703 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73532 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75586 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74643 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn Waiting for MDT destroys to complete PASS 77kg (92s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 00:40:00 (1761280800) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (4s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 00:40:04 (1761280804) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg128-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.58254 s, 293 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg128-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0082 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (51s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 00:40:55 (1761280855) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0527965 s, 318 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142596 s, 73.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269697 s, 38.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269492 s, 38.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293192 s, 35.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370103 s, 28.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417763 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341878 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260724 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225246 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03404 s, 30.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403071 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296965 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347873 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240553 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453565 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303206 s, 34.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff931f03c9b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163241 s, 64.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222684 s, 47.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247751 s, 42.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266565 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014801 s, 70.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299385 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147205 s, 71.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254943 s, 41.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119375 s, 87.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264404 s, 39.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023833 s, 44.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289568 s, 36.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237856 s, 44.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176859 s, 59.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00567786 s, 185 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139664 s, 75.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff931f03c9b800.lock_unused_count=1 removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg128-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg128-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90899 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97148 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0634017 s, 265 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161537 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280938 s, 37.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324999 s, 32.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364675 s, 28.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403562 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384652 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319245 s, 32.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538957 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485971 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317413 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039546 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417359 s, 25.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407156 s, 25.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221686 s, 47.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478663 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384352 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100793 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195737 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167691 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019123 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311753 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212168 s, 49.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210315 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265733 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00981445 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307134 s, 34.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289699 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121994 s, 86.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168475 s, 62.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00886737 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460592 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0062563 s, 168 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg128-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg128-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90102 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97209 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (47s) == sanityn test 77o: Changing rank should not panic ====== 00:41:42 (1761280902) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 00:41:46 (1761280906) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (38s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 00:42:24 (1761280944) mds.MDS.mdt.nrs_policies=tbf oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg128-client: oleg128-server: ssh exited with exit code 36 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 oleg128-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg128-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (14s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 00:42:38 (1761280958) striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d77r.sanityn mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 339844 kill md_thread xargs PASS 77r (43s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 00:43:21 (1761281001) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 00:43:24 (1761281004) striped dir -i1 -c2 -H crush2 /mnt/lustre/d79.sanityn File: /mnt/lustre/d79.sanityn Size: 8192 Blocks: 16 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798763921 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 00:43:25.000000000 -0400 Modify: 2025-10-24 00:43:25.000000000 -0400 Change: 2025-10-24 00:43:25.000000000 -0400 Birth: 2025-10-24 00:43:25.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 00:43:37 (1761281017) total: 10 open/close in 0.03 seconds: 374.18 ops/second multiop /mnt/lustre2/d80a.sanityn/f80a.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Finish migration, then checking.. PASS 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 00:43:40 (1761281020) start migration thread 349406 accessing the migrating directory for 1 minute... concurrent access 201 failures, 447 successes /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 349406 Killed ( while true; do mdt_idx=$((RANDOM % MDSCOUNT)); $LFS migrate -m $mdt_idx $migdir1 &> /dev/null; done ) (wd: ~) PASS 80b (114s) == sanityn test 81a: rename and stat under striped directory ========================================================== 00:45:34 (1761281134) stat: cannot statx 'd0/0': No such file or directory File: d0/0 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289283068 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 00:45:34.000000000 -0400 Modify: 2025-10-24 00:45:34.000000000 -0400 Change: 2025-10-24 00:45:34.000000000 -0400 Birth: 2025-10-24 00:45:34.000000000 -0400 PASS 81a (2s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 00:45:36 (1761281136) total: 2500 open/close in 6.38 seconds: 391.83 ops/second /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 351540 Killed ( while true; do touch $DIR1/$tdir; done ) (wd: ~) PASS 81b (45s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 00:46:21 (1761281181) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 00:46:22 (1761281182) total: 2500 open/close in 7.73 seconds: 323.44 ops/second total: 2500 open/close in 7.52 seconds: 332.33 ops/second snapshot_time 1761281222.057817467 secs.nsecs start_time 1761281183.886179872 secs.nsecs elapsed_time 38.171637595 secs.nsecs open 2505 samples [usecs] 64 4594 874270 367336274 close 2504 samples [usecs] 44 327 245526 27370902 mknod 2501 samples [usecs] 150 4523 736102 269951988 unlink 2500 samples [usecs] 135 10587 600292 528498662 mkdir 2 samples [usecs] 530 2024 2554 4377476 rmdir 2 samples [usecs] 1208 7744 8952 61428800 rename 2500 samples [usecs] 131 6873 2201696 3288142684 getattr 3764 samples [usecs] 0 223 57294 1187114 statfs 7 samples [usecs] 8 18 86 1146 parallel_rename_file 2500 samples [usecs] 131 6873 2201696 3288142684 crossdir_rename 2500 samples [usecs] 131 6873 2201696 3288142684 snapshot_time 1761281222.057871957 secs.nsecs start_time 1761281183.886240219 secs.nsecs elapsed_time 38.171631738 secs.nsecs open 2504 samples [usecs] 51 1238 849043 318916513 close 2504 samples [usecs] 48 4342 242924 44752554 mknod 2500 samples [usecs] 151 1170 714703 227438589 unlink 2500 samples [usecs] 130 5619 565752 265072020 mkdir 2 samples [usecs] 531 6594 7125 43762797 rmdir 2 samples [usecs] 1273 12792 14065 165255793 rename 2500 samples [usecs] 135 4966 2169285 3118153491 getattr 3757 samples [usecs] 5 888 63153 2113793 statfs 7 samples [usecs] 7 19 90 1316 parallel_rename_file 2500 samples [usecs] 135 4966 2169285 3118153491 crossdir_rename 2500 samples [usecs] 135 4966 2169285 3118153491 PASS 81d (42s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 00:47:04 (1761281224) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.8189 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.8189 PASS 82 (2s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 00:47:06 (1761281226) start pid 365840 to create/unlink striped directory start pid 365841 to stat striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5797: 365840 Terminated ( cd $DIR1; while true; do $LFS mkdir -i1 -c2 $tdir; rmdir $tdir; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5797: 365841 Terminated ( cd $DIR2; while true; do stat $tdir > /dev/null 2>&1; done ) PASS 83 (122s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 00:49:08 (1761281348) fail_loc=0x8000060b PASS 84 (8s) == sanityn test 85: Lustre API root cache race =========== 00:49:16 (1761281356) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1761281357 Finishing test test1 at 1761281357 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 00:49:19 (1761281359) start pid 378406 to open/create under striped directory start pid 378407 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5864: 378406 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; touch $tdir/f{0..3} > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5864: 378407 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 90 (183s) == sanityn test 91: chmod and unlink striped directory === 00:52:22 (1761281542) start pid 392507 to chmod striped directory start pid 392508 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5901: 392507 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; chmod go+w $tdir > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5901: 392508 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 91 (183s) == sanityn test 92: create remote directory under orphan directory ========================================================== 00:55:25 (1761281725) fail_loc=0x1408 mkdir: cannot create directory '/mnt/lustre2/d92.sanityn/dir': No such file or directory lfs setdirstripe: cannot resolve path '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory (2) lfs setdirstripe: '/mnt/lustre2/d92.sanityn/remote_dir' is not on a Lustre filesystem: No such file or directory (2) lfs setdirstripe: cannot create dir '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory fail_loc=0 PASS 92 (2s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 00:55:27 (1761281727) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6729 0x1a49 0x2c0000401 0 6747 0x1a5b 0x280000401 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6730 0x1a4a 0x2c0000401 0 6748 0x1a5c 0x280000401 rc1=0 and rc2=0 PASS 93 (9s) == sanityn test 94: signal vs CP callback race =========== 00:55:36 (1761281736) osc.lustre-OST0000-osc-ffff931f03c9b800.idle_timeout=0 osc.lustre-OST0000-osc-ffff931f20ea3000.idle_timeout=0 osc.lustre-OST0001-osc-ffff931f03c9b800.idle_timeout=0 osc.lustre-OST0001-osc-ffff931f20ea3000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00752262 s, 1.1 MB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.0969 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5988: 413397 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00919329 s, 446 kB/s osc.lustre-OST0000-osc-ffff931f03c9b800.idle_timeout=debug osc.lustre-OST0000-osc-ffff931f20ea3000.idle_timeout=debug osc.lustre-OST0001-osc-ffff931f03c9b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff931f20ea3000.idle_timeout=debug PASS 94 (10s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 00:55:46 (1761281746) llite.lustre-ffff931f03c9b800.fast_read=0 llite.lustre-ffff931f20ea3000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.00777094 s, 270 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff931f03c9b800.fast_read=1 llite.lustre-ffff931f20ea3000.fast_read=1 PASS 95a (15s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 00:56:01 (1761281761) llite.lustre-ffff931f03c9b800.fast_read=0 llite.lustre-ffff931f20ea3000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00147783 s, 8.3 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff931f03c9b800.fast_read=1 llite.lustre-ffff931f20ea3000.fast_read=1 PASS 95b (9s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 00:56:10 (1761281770) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (2s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 00:56:12 (1761281772) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00176632 s, 2.3 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (3s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 00:56:15 (1761281775) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0220452 s, 95.1 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (3s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 00:56:18 (1761281778) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0205589 s, 102 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (3s) == sanityn test 100e: DoM: read on open and file size ==== 00:56:21 (1761281781) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00159736 s, 7.5 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 00:56:23 (1761281783) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000559395 s, 7.3 MB/s PASS 101a (3s) == sanityn test 101b: Discard DoM data on rename ========= 00:56:26 (1761281786) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000491423 s, 8.3 MB/s PASS 101b (3s) == sanityn test 101c: Discard DoM data on close-unlink === 00:56:29 (1761281789) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000549628 s, 7.5 MB/s PASS 101c (4s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 00:56:33 (1761281793) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 00 02 00 00 00 1d 32 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000402:0x321d:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.remote fh_bytes: 32 fh_type: 151 fh_data: 05 04 00 40 02 00 00 00 1d 36 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000405:0x361d:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.remote/subdir fh_bytes: 32 fh_type: 151 fh_data: 05 04 00 40 02 00 00 00 1e 36 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000405:0x361e:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.stripe fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 00 02 00 00 00 1e 32 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000402:0x321e:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.stripe/subdir fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 00 02 00 00 00 1f 32 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000402:0x321f:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! PASS 102 (4s) == sanityn test 103: Test size correctness with lockahead ========================================================== 00:56:37 (1761281797) striped dir -i1 -c2 -H all_char /mnt/lustre/d103.sanityn fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1761281798 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1761281801 Finishing test test23 at 1761281801 PASS 103 (8s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 00:56:45 (1761281805) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00399127 s, 257 kB/s oleg128-server: debugfs 1.47.1-wc2 (08-Nov-2024) STAT a|m|ctime 1761281807 1761281807 1761281807 MDT a|m|ctime 1761281807 1761281807 1761281807 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00219419 s, 467 kB/s oleg128-server: debugfs 1.47.1-wc2 (08-Nov-2024) STAT a|m|ctime 1761281807 1761281809 1761281809 MDT a|m|ctime 1761281807 1761281809 1761281809 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0107633 s, 95.1 kB/s oleg128-server: debugfs 1.47.1-wc2 (08-Nov-2024) STAT a|m|ctime 1761281807 1761281814 1761281814 MDT a|m|ctime 1761281807 1761281814 1761281814 oleg128-server: debugfs 1.47.1-wc2 (08-Nov-2024) oleg128-server: debugfs 1.47.1-wc2 (08-Nov-2024) STAT a|m|ctime 1761281807 1761281814 1761281814 MDT a|m|ctime 1761281807 1761281814 1761281814 oleg128-server: debugfs 1.47.1-wc2 (08-Nov-2024) STAT a|m|ctime 1761281807 1761281814 1761281823 MDT a|m|ctime 1761281807 1761281814 1761281823 PASS 104 (20s) == sanityn test 105: Glimpse and lock cancel race ======== 00:57:05 (1761281825) striped dir -i1 -c2 -H all_char /mnt/lustre/d105.sanityn fail_loc=0x416 PASS 105 (18s) == sanityn test 106a: Verify the btime via statx() ======= 00:57:23 (1761281843) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00345284 s, 297 kB/s oleg128-server: debugfs 1.47.1-wc2 (08-Nov-2024) Inode: 5235 Type: regular Mode: 0644 Flags: 0x80000 Generation: 2895294638 Version: 0x00000002:000183a4 User: 0 Group: 0 Project: 0 Size: 0 File ACL: 0 Links: 1 Blockcount: 0 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x68fb0734:00000000 -- Fri Oct 24 00:57:24 2025 atime: 0x68fb0734:00000000 -- Fri Oct 24 00:57:24 2025 mtime: 0x68fb0734:00000000 -- Fri Oct 24 00:57:24 2025 crtime: 0x68fb0734:bc1cbd00 -- Fri Oct 24 00:57:24 2025 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x200000402:0x3222:0x0] compat=0 incompat=0 trusted.lov (56) = d0 0b d1 0b 01 00 00 00 22 32 00 00 00 00 00 00 02 04 00 00 02 00 00 00 00 00 40 00 01 00 00 00 01 04 00 80 02 00 00 00 61 1a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 user.job (4) = "dd.0" linkea: idx=0 parent=[0x200000007:0x1:0x0] name='f106a.sanityn' trusted.som (24) = 04 00 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 EXTENTS: PASS 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 00:57:27 (1761281847) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00385771 s, 265 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 8 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205289292323 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 00:57:27.000000000 -0400 Modify: 2025-10-24 00:57:27.000000000 -0400 Change: 2025-10-24 00:57:27.000000000 -0400 Birth: 2025-10-24 00:57:28.000000000 -0400 /mnt/lustre/f106b.sanityn 144115205289292323 -rw-r--r-- 1 0 0 1761281848 1761281847 1761281847 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 8 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205289292323 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 00:57:27.000000000 -0400 Modify: 2025-10-24 00:57:27.000000000 -0400 Change: 2025-10-24 00:57:27.000000000 -0400 Birth: 2025-10-24 00:57:28.000000000 -0400 1761281847 1024 8 PASS 106b (3s) == sanityn test 106c: Verify statx attributes mask ======= 00:57:30 (1761281850) PASS 106c (3s) == sanityn test 107a: Basic grouplock conflict =========== 00:57:33 (1761281853) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0553383 s, 189 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6737 0x1a51 0x2c0000401 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 First grouplock blocks second one PASS 107a (5s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 00:57:38 (1761281858) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0728866 s, 144 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6755 0x1a63 0x280000401 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (10s) == sanityn test 108a: lseek: parallel updates ============ 00:57:48 (1761281868) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00239407 s, 4.2 MB/s llite.lustre-ffff931f03c9b800.hybrid_io=0 llite.lustre-ffff931f20ea3000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.1978 s, 2.0 MB/s PASS 108a (8s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 00:57:56 (1761281876) 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 192.168.201.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 PASS 109 (564s) == sanityn test 110: do not grant another lock on resend ========================================================== 01:07:20 (1761282440) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934275 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 01:07:21.000000000 -0400 Modify: 2025-10-24 01:07:21.000000000 -0400 Change: 2025-10-24 01:07:21.000000000 -0400 Birth: 2025-10-24 01:07:21.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg128-client.virtnet oleg128-client.virtnet PASS 110 (65s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 01:08:25 (1761282505) fail_loc=0x8000018a /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6784: 486131 Terminated $MULTIOP $DIR2/$tdir/mdt0dir/foodir/file2 Ow4096_c PASS 111 (6s) == sanityn test 112: update max-inherit in default LMV === 01:08:31 (1761282511) lmv.lustre-clilmv-ffff931f10cfd000.qos_threshold_rr=90 lmv.lustre-clilmv-ffff931f33b9e800.qos_threshold_rr=90 PASS 112 (5s) == sanityn test 113: check servers of specified fs ======= 01:08:36 (1761282516) lustre-OST0000-osc-ffff931f33b9e800 active. lustre-OST0001-osc-ffff931f33b9e800 active. lustre-OST0000-osc-ffff931f33b9e800 active. lustre-OST0001-osc-ffff931f33b9e800 active. MGC192.168.201.128@tcp active. PASS 113 (3s) == sanityn test 114: implicit default LMV inherit ======== 01:08:39 (1761282519) /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 PASS 114 (10s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 01:08:49 (1761282529) total: 3000 create in 4.95 seconds: 606.24 ops/second oleg128-server: debugfs 1.47.1-wc2 (08-Nov-2024) oleg128-server: htree_dump_int_block: Ext2 directory block not found while calculating checksum for htree root oleg128-server: USE: f1151 f1785 fail_loc=0x80002401 fail_val=5 mkdir: cannot create directory '/mnt/lustre/d115.sanityn/f1785': File exists PASS 115 (14s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 01:09:03 (1761282543) llite.lustre-ffff931f10cfd000.intent_mkdir=1 llite.lustre-ffff931f33b9e800.intent_mkdir=1 MD layout /mnt/lustre/d116.sanityn: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000bd0:0x99c:0x0] 1 [0x240000404:0x987:0x0] mkdir /mnt/lustre/d116.sanityn/tdir0 setdirstripe -D -i1 /mnt/lustre2/d116.sanityn/tdir0 mkdir /mnt/lustre/d116.sanityn/tdir0/tdir11 lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none mdtidx FID[seq:oid:ver] lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none mdtidx FID[seq:oid:ver] PASS 116 (3s) == sanityn test 117a: TCU: Init and enable Trash Can on MDTs ========================================================== 01:09:06 (1761282546) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 PASS 117a (5s) == sanityn test 117b: Move regular file and empty dir into trash can dir ========================================================== 01:09:11 (1761282551) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain llite.lustre-ffff931f10cfd000.trash_file_read=1 llite.lustre-ffff931f33b9e800.trash_file_read=1 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0965785 s, 43.4 MB/s /mnt/lustre/.lustre/trash: MDT0000 MDT0001 /mnt/lustre/.lustre/trash/MDT0000: 0x200000bd1:0xbcc:0x0 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbcc:0x0: tdir1 tdir2 tf1 tf2 tf3 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbcc:0x0/tdir1: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbcc:0x0/tdir2: /mnt/lustre/.lustre/trash/MDT0001: llite.lustre-ffff931f10cfd000.trash_file_read=0 llite.lustre-ffff931f33b9e800.trash_file_read=0 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117b (7s) == sanityn test 117c: Move deleted tree with multiple levels into trash ========================================================== 01:09:19 (1761282559) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain /mnt/lustre/.lustre/trash/MDT0000: 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbd2:0x0: tdir1 tdir2 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbd2:0x0/tdir1: tdir11 tdir12 tfile13 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbd2:0x0/tdir1/tdir11: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbd2:0x0/tdir1/tdir12: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbd2:0x0/tdir2: mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117c (7s) == sanityn test 117d: Per-User Trash can Type testing ==== 01:09:26 (1761282566) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid Trash Can Type: uid uid running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mkdir] [/mnt/lustre/d117d.sanityn/tdir2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mkdir] [/mnt/lustre/d117d.sanityn/tdir2/tdir21] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mkdir] [/mnt/lustre/d117d.sanityn/tdir2/tdir22] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d117d.sanityn/tdir2/tfile23] uid1: 0 uid2: 500 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: 0x200000bd1:0xbdb:0x0 /mnt/lustre/.lustre/trash/MDT0000/0/0x200000bd1:0xbdb:0x0: tdir11 tdir12 tfile13 /mnt/lustre/.lustre/trash/MDT0000/0/0x200000bd1:0xbdb:0x0/tdir11: /mnt/lustre/.lustre/trash/MDT0000/0/0x200000bd1:0xbdb:0x0/tdir12: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/500: 0x200000bd1:0xbdf:0x0 /mnt/lustre/.lustre/trash/MDT0000/500/0x200000bd1:0xbdf:0x0: tdir21 tdir22 tfile23 /mnt/lustre/.lustre/trash/MDT0000/500/0x200000bd1:0xbdf:0x0/tdir21: /mnt/lustre/.lustre/trash/MDT0000/500/0x200000bd1:0xbdf:0x0/tdir22: mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117d (7s) == sanityn test 117e: Undeleted dir in trash should keep its original xattrs ========================================================== 01:09:33 (1761282573) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe3:0x0: tdir1 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe3:0x0/tdir1: tdir11 tfile12 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe3:0x0/tdir1/tdir11: /mnt/lustre/.lustre/trash/MDT0000/500: trusted.name11="value11" trusted.name12="value12" trusted.name1="value1" mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117e (8s) == sanityn test 117f: Uncache the dentry under the trash dir ========================================================== 01:09:41 (1761282581) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe7:0x0: tdir1 tfile3 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe7:0x0/tdir1: /mnt/lustre/.lustre/trash/MDT0000/500: tdir1 tfile3 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe7:0x0: tdir1 tdir2 tfile3 tfile4 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe7:0x0/tdir1: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe7:0x0/tdir2: /mnt/lustre/.lustre/trash/MDT0000/500: tdir2 tfile4 -u-------------------- /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe7:0x0 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe7:0x0: -u------------e------- /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe7:0x0/tfile4 -u------------e------- /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe7:0x0/tfile3 -u-------------------- /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe7:0x0/tdir1 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe7:0x0/tdir1: -u-------------------- /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe7:0x0/tdir2 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe7:0x0/tdir2: -u-------------------- /mnt/lustre/.lustre/trash/MDT0000/0 /mnt/lustre/.lustre/trash/MDT0000/0: -u-------------------- /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbcc:0x0 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbcc:0x0: -u-------------------- /mnt/lustre/.lustre/trash/MDT0000/500 /mnt/lustre/.lustre/trash/MDT0000/500: -u-------------------- /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe3:0x0 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe3:0x0: -u-------------------- /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbd2:0x0 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbd2:0x0: mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117f (7s) == sanityn test 117g: Access .Trash for a non-striped directory ========================================================== 01:09:48 (1761282588) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbec:0x0: tdir1 tfile3 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbec:0x0/tdir1: /mnt/lustre/.lustre/trash/MDT0000/500: ==> check '.Trash': tdir1 tfile3 <== tdir1 tfile3 tdir1 tfile3 tdir21 tfile22 tdir2 tdir21 tfile22 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117g (8s) == sanityn test 117h: Permission testing of various operations for files in Trash Can ========================================================== 01:09:56 (1761282596) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain Trash list: ===> /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0: tdir1 tfile3 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0/tdir1: /mnt/lustre/.lustre/trash/MDT0000/500: <=== touch: setting times of '/mnt/lustre/.lustre/trash/f117h.sanityn.crfail': No such file or directory mkdir: cannot create directory '/mnt/lustre/.lustre/trash/f117h.sanityn.crfail': Operation not permitted touch: setting times of '/mnt/lustre/.lustre/trash/MDT0000/f117h.sanityn.crfail': No such file or directory mkdir: cannot create directory '/mnt/lustre/.lustre/trash/MDT0000/f117h.sanityn.crfail': Operation not permitted rmdir: failed to remove '/mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0': Operation not permitted touch: setting times of '/mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0/f117h.sanityn.crfail': No such file or directory mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/f117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/f117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0/f117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/d117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/d117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0/d117h.sanityn.mvfail': Operation not permitted chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0//tfile3 chmod: changing permissions of '/mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0//tfile3': Bad file descriptor truncate: cannot truncate '/mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0//tfile3' to length 4096: Bad file descriptor chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0//tdir1 chmod: changing permissions of '/mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0//tdir1': Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0//tfile3: Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0//tfile3: Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0//tdir1: Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0//tdir1: Bad file descriptor open: Bad file descriptor open: Bad file descriptor read: Operation not permitted mmap: Operation not permitted open: Is a directory open: Is a directory mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117h (8s) == sanityn test 117i: Forbid to create a file with name .Trash ========================================================== 01:10:04 (1761282604) mkdir: cannot create directory '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument touch: setting times of '/mnt/lustre/d117i.sanityn/.Trash': No such file or directory open(O_RDWR|O_CREAT): Invalid argument mcreate: cannot create `/mnt/lustre/d117i.sanityn/.Trash' with mode 0100644: Invalid argument ln: failed to create hard link '/mnt/lustre/d117i.sanityn/.Trash' => '/mnt/lustre/d117i.sanityn/f117i.sanityn': Invalid argument ln: failed to create symbolic link '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument mv: cannot move '/mnt/lustre/d117i.sanityn/f117i.sanityn' to a subdirectory of itself, '/mnt/lustre/d117i.sanityn/.Trash' mv: cannot move '/mnt/lustre/d117i.sanityn/f117i.sanityn' to a subdirectory of itself, '/mnt/lustre/.Trash' ln: failed to create symbolic link '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument mv: cannot move '/mnt/lustre/d117i.sanityn/d117i.sanityn' to a subdirectory of itself, '/mnt/lustre/d117i.sanityn/.Trash' mv: cannot move '/mnt/lustre/d117i.sanityn/d117i.sanityn' to a subdirectory of itself, '/mnt/lustre/.Trash' PASS 117i (4s) == sanityn test 117j: Move back file from Trash Can ====== 01:10:08 (1761282608) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain tfile.1 -u------------e------- /mnt/lustre/d117j.sanityn/.Trash/tfile.1 --------------e------- /mnt/lustre/d117j.sanityn/tfile.1 List /mnt/lustre/d117j.sanityn/.Trash /mnt/lustre/d117j.sanityn/.Trash: tdir1 /mnt/lustre/d117j.sanityn/.Trash/tdir1: tdir11 tfile12 tfile13 /mnt/lustre/d117j.sanityn/.Trash/tdir1/tdir11: -u-------------------- /mnt/lustre/d117j.sanityn/.Trash/tdir1 /mnt/lustre/d117j.sanityn/.Trash/tdir1: -u-------------------- /mnt/lustre/d117j.sanityn/.Trash/tdir1/tdir11 /mnt/lustre/d117j.sanityn/.Trash/tdir1/tdir11: -u------------e------- /mnt/lustre/d117j.sanityn/.Trash/tdir1/tfile12 -u------------e------- /mnt/lustre/d117j.sanityn/.Trash/tdir1/tfile13 === DONE === mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117j (7s) == sanityn test 117k: Unrm file from Trash Can =========== 01:10:15 (1761282615) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain ==== DONE ==== mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117k (9s) == sanityn test 117l: Clean up files in Trash Can via command 'lfs trash clean' ========================================================== 01:10:24 (1761282624) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain Phase 0/PR: remove dir 1:0x200000bd1:0xbe7:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbe7:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbe7:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbe7:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc12:0x0 from Trash Can Phase 0/PR: remove dir 2:tdir1 from Trash Can Phase 0/PR: remove dir 3:tdir11 from Trash Can Phase 2/PO: remove dir 3:tdir11 from Trash Can Phase 1/EN: remove dir 3:tdir11 from Trash Can Phase 1/EN: remove file 3:tfile12 from Trash Can Phase 1/EN: remove file 3:tfile13 from Trash Can Phase 2/PO: remove dir 2:tdir1 from Trash Can Phase 1/EN: remove dir 2:tdir1 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc12:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc12:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc12:0x0 from Trash Can Phase 0/PR: remove dir 1:0 from Trash Can Phase 2/PO: remove dir 1:0 from Trash Can Phase 1/EN: remove dir 1:0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc0c:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc0c:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc0c:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc0c:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc06:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc06:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc06:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc06:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbcc:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbcc:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbcc:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbcc:0x0 from Trash Can Phase 0/PR: remove dir 1:500 from Trash Can Phase 2/PO: remove dir 1:500 from Trash Can Phase 1/EN: remove dir 1:500 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbe3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbe3:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbe3:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbe3:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbf3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbf3:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbf3:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbf3:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbec:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbec:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbec:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbec:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbd2:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbd2:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbd2:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbd2:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbe7:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbe7:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbe7:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbe7:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc12:0x0 from Trash Can Phase 0/PR: remove dir 2:tdir1 from Trash Can Phase 0/PR: remove dir 3:tdir11 from Trash Can Phase 2/PO: remove dir 3:tdir11 from Trash Can Phase 1/EN: remove dir 3:tdir11 from Trash Can Phase 1/EN: remove file 3:tfile12 from Trash Can Phase 1/EN: remove file 3:tfile13 from Trash Can Phase 2/PO: remove dir 2:tdir1 from Trash Can Phase 1/EN: remove dir 2:tdir1 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc12:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc12:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc12:0x0 from Trash Can Phase 0/PR: remove dir 1:0 from Trash Can Phase 2/PO: remove dir 1:0 from Trash Can Phase 1/EN: remove dir 1:0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc0c:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc0c:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc0c:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc0c:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc06:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc06:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc06:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc06:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbcc:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbcc:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbcc:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbcc:0x0 from Trash Can Phase 0/PR: remove dir 1:500 from Trash Can Phase 2/PO: remove dir 1:500 from Trash Can Phase 1/EN: remove dir 1:500 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc1c:0x0 from Trash Can Phase 1/EN: remove file 2:tfile22 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc1c:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc1c:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc1c:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbe3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbe3:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbe3:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbe3:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbf3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbf3:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbf3:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbf3:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbec:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbec:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbec:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbec:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbd2:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbd2:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbd2:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbd2:0x0 from Trash Can mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117l (7s) == sanityn test 117m: Empty Trash Can via the command 'lfs trash clean' ========================================================== 01:10:32 (1761282632) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain /mnt/lustre/.lustre/trash: MDT0000 MDT0001 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc1f:0x0: tdir1 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc1f:0x0/tdir1: tdir11 tfile12 tfile13 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc1f:0x0/tdir1/tdir11: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc25:0x0: tfile22 /mnt/lustre/.lustre/trash/MDT0000/500: /mnt/lustre/.lustre/trash/MDT0001: Phase 0/PR: remove dir 1:0x200000bd1:0xbe7:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbe7:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbe7:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbe7:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc12:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc12:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc12:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc12:0x0 from Trash Can Phase 0/PR: remove dir 1:0 from Trash Can Phase 2/PO: remove dir 1:0 from Trash Can Phase 1/EN: remove dir 1:0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc0c:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc0c:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc0c:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc0c:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc1f:0x0 from Trash Can Phase 0/PR: remove dir 2:tdir1 from Trash Can Phase 0/PR: remove dir 3:tdir11 from Trash Can Phase 2/PO: remove dir 3:tdir11 from Trash Can Phase 1/EN: remove dir 3:tdir11 from Trash Can Phase 1/EN: remove file 3:tfile12 from Trash Can Phase 1/EN: remove file 3:tfile13 from Trash Can Phase 2/PO: remove dir 2:tdir1 from Trash Can Phase 1/EN: remove dir 2:tdir1 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc1f:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc1f:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc1f:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc06:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc06:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc06:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc06:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbcc:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbcc:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbcc:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbcc:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc25:0x0 from Trash Can Phase 1/EN: remove file 2:tfile22 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc25:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc25:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc25:0x0 from Trash Can Phase 0/PR: remove dir 1:500 from Trash Can Phase 2/PO: remove dir 1:500 from Trash Can Phase 1/EN: remove dir 1:500 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc1c:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc1c:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc1c:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc1c:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbe3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbe3:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbe3:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbe3:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbf3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbf3:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbf3:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbf3:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbec:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbec:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbec:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbec:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbd2:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbd2:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbd2:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbd2:0x0 from Trash Can mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117m (7s) == sanityn test 117n: Update LinkEA accordingly when moving file into Trash Can ========================================================== 01:10:40 (1761282640) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain /mnt/lustre/d117n.sanityn/.Trash//tfile.1: count 1 0: pfid [0x200000bd0:0x9b6:0x0], name 'tfile.1' /mnt/lustre/d117n.sanityn/.Trash//tfile.1: count 1 0: pfid [0x200000bd0:0x9b6:0x0], name 'tfile.1' /mnt/lustre/d117n.sanityn/.Trash//tfile.1: count 1 0: pfid [0x200000bd0:0x9b6:0x0], name 'tfile.1' /mnt/lustre/d117n.sanityn/.Trash//tfile.1: count 1 0: pfid [0x200000bd0:0x9b6:0x0], name 'tfile.1' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tdir11: count 1 0: pfid [0x200000bd0:0x9b7:0x0], name 'tdir11' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tdir11: count 1 0: pfid [0x200000bd0:0x9b7:0x0], name 'tdir11' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tdir11: count 1 0: pfid [0x200000bd0:0x9b7:0x0], name 'tdir11' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tdir11: count 1 0: pfid [0x200000bd0:0x9b7:0x0], name 'tdir11' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile12: count 1 0: pfid [0x200000bd0:0x9b7:0x0], name 'tfile12' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile12: count 1 0: pfid [0x200000bd0:0x9b7:0x0], name 'tfile12' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile12: count 1 0: pfid [0x200000bd0:0x9b7:0x0], name 'tfile12' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile12: count 1 0: pfid [0x200000bd0:0x9b7:0x0], name 'tfile12' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile13: count 1 0: pfid [0x200000bd0:0x9b7:0x0], name 'tfile13' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile13: count 1 0: pfid [0x200000bd0:0x9b7:0x0], name 'tfile13' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile13: count 1 0: pfid [0x200000bd0:0x9b7:0x0], name 'tfile13' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile13: count 1 0: pfid [0x200000bd0:0x9b7:0x0], name 'tfile13' /mnt/lustre/d117n.sanityn/.Trash//tdir1: count 1 0: pfid [0x200000bd0:0x9b6:0x0], name 'tdir1' /mnt/lustre/d117n.sanityn/.Trash//tdir1: count 1 0: pfid [0x200000bd0:0x9b6:0x0], name 'tdir1' /mnt/lustre/d117n.sanityn/.Trash//tdir1: count 1 0: pfid [0x200000bd0:0x9b6:0x0], name 'tdir1' /mnt/lustre/d117n.sanityn/.Trash//tdir1: count 1 0: pfid [0x200000bd0:0x9b6:0x0], name 'tdir1' /mnt/lustre/d117n.sanityn/tdir2/.Trash//tfile22: count 1 0: pfid [0x200000bd0:0x9b8:0x0], name 'tfile22' /mnt/lustre/d117n.sanityn/tdir2/.Trash//tfile22: count 1 0: pfid [0x200000bd0:0x9b8:0x0], name 'tfile22' /mnt/lustre/d117n.sanityn/tdir2/.Trash//tfile22: count 1 0: pfid [0x200000bd0:0x9b8:0x0], name 'tfile22' /mnt/lustre/d117n.sanityn/tdir2/.Trash//tfile22: count 1 0: pfid [0x200000bd0:0x9b8:0x0], name 'tfile22' mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117n (9s) == sanityn test 117o: Set/Clear NOTRASH flag on file ===== 01:10:48 (1761282648) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain Set flags 0X80000/0X80040 OK: rc=0 file: /mnt/lustre/d117o.sanityn/tfile.1,flags: 0X80040 notrash, uid: 0 gid: 0 projid: 0 timestamp: 1761282651 file: /mnt/lustre/d117o.sanityn/tfile.1,flags: 0X80040 notrash, uid: 0 gid: 0 projid: 0 timestamp: 1761282651 file: /mnt/lustre/d117o.sanityn/tdir1,flags: 0X40 notrash, uid: 0 gid: 0 projid: 0 timestamp: 1761282652 file: /mnt/lustre/d117o.sanityn/tdir1/tdir11,flags: 0X40 notrash, uid: 0 gid: 0 projid: 0 timestamp: 1761282652 file: /mnt/lustre/d117o.sanityn/tdir1,flags: 0X40 notrash, uid: 0 gid: 0 projid: 0 timestamp: 1761282652 file: /mnt/lustre/d117o.sanityn/tdir1/tdir11,flags: 0X7F9E00000040 notrash, uid: 0 gid: 0 projid: 0 timestamp: 1761282652 file: /mnt/lustre/d117o.sanityn/tdir1/tfile12,flags: 0X7F9E00080040 notrash, uid: 0 gid: 0 projid: 0 timestamp: 1761282652 lsattr /mnt/lustre/d117o.sanityn/tdir1: ------d--------------- /mnt/lustre/d117o.sanityn/tdir1/tdir11 ------d-------e------- /mnt/lustre/d117o.sanityn/tdir1/tfile12 lsattr /mnt/lustre/d117o.sanityn/tdir1/tdir11: lsattr /mnt/lustre/d117o.sanityn/tdir1/tfile12: ------d-------e------- /mnt/lustre/d117o.sanityn/tdir1/tfile12 file: /mnt/lustre/d117o.sanityn/tdir1/tdir11,flags: 0X40 notrash, uid: 0 gid: 0 projid: 0 timestamp: 1761282652 file: /mnt/lustre/d117o.sanityn/tdir1/tfile12,flags: 0X80040 notrash, uid: 0 gid: 0 projid: 0 timestamp: 1761282652 file: /mnt/lustre/d117o.sanityn/tdir1/tfile13,flags: 0X80040 notrash, uid: 0 gid: 0 projid: 0 timestamp: 1761282652 Set flags 0X80040/0X80000 OK: rc=0 chattr 1.47.1-wc2 (08-Nov-2024) Flags of /mnt/lustre/d117o.sanityn/tdir1 set as ---------------------- file: /mnt/lustre/d117o.sanityn/tdir1/tfile13,flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1761282652 tfile13 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117o (8s) == sanityn test 117p: Trash Can handling for remote dir/file ========================================================== 01:10:56 (1761282656) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain /mnt/lustre/d117p.sanityn/d117p.sanityn.1/.Trash/f117p.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6761 0x1a69 0x2c0000401 /mnt/lustre/.lustre/trash: MDT0000 MDT0001 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/trash/MDT0000/500: /mnt/lustre/.lustre/trash/MDT0001: 0x240000406:0xc:0x0 /mnt/lustre/.lustre/trash/MDT0001/0x240000406:0xc:0x0: f117p.sanityn /mnt/lustre/.lustre/trash: MDT0000 MDT0001 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc37:0x0: d117p.sanityn.01 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc37:0x0/d117p.sanityn.01: /mnt/lustre/.lustre/trash/MDT0000/500: /mnt/lustre/.lustre/trash/MDT0001: 0x240000406:0xc:0x0 /mnt/lustre/.lustre/trash/MDT0001/0x240000406:0xc:0x0: f117p.sanityn mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117p (8s) == sanityn test 117q: Access Trash Can via '.Trash' for striped directory ========================================================== 01:11:04 (1761282664) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain #0 -> 0x200000bd0:0x9bb:0x0 #1 -> 0x240000404:0x988:0x0 file.1 file.2 Creating 100 files under striped directory: /mnt/lustre/d117q.sanityn/striped_dir total: 100 open/close in 0.60 seconds: 167.10 ops/second f117q.sanityn0 -> 0 f117q.sanityn1 -> 0 f117q.sanityn2 -> 0 f117q.sanityn3 -> 0 f117q.sanityn4 -> 0 f117q.sanityn5 -> 0 f117q.sanityn6 -> 0 f117q.sanityn7 -> 0 f117q.sanityn8 -> 0 f117q.sanityn9 -> 0 f117q.sanityn10 -> 0 f117q.sanityn11 -> 0 f117q.sanityn12 -> 0 f117q.sanityn13 -> 0 f117q.sanityn14 -> 1 f117q.sanityn15 -> 1 f117q.sanityn16 -> 0 f117q.sanityn17 -> 1 f117q.sanityn18 -> 0 f117q.sanityn19 -> 1 f117q.sanityn20 -> 1 f117q.sanityn21 -> 1 f117q.sanityn22 -> 1 f117q.sanityn23 -> 1 f117q.sanityn24 -> 1 f117q.sanityn25 -> 0 f117q.sanityn26 -> 1 f117q.sanityn27 -> 0 f117q.sanityn28 -> 1 f117q.sanityn29 -> 1 f117q.sanityn30 -> 1 f117q.sanityn31 -> 1 f117q.sanityn32 -> 0 f117q.sanityn33 -> 1 f117q.sanityn34 -> 0 f117q.sanityn35 -> 0 f117q.sanityn36 -> 1 f117q.sanityn37 -> 0 f117q.sanityn38 -> 1 f117q.sanityn39 -> 0 f117q.sanityn40 -> 1 f117q.sanityn41 -> 0 f117q.sanityn42 -> 1 f117q.sanityn43 -> 1 f117q.sanityn44 -> 0 f117q.sanityn45 -> 0 f117q.sanityn46 -> 0 f117q.sanityn47 -> 1 f117q.sanityn48 -> 1 f117q.sanityn49 -> 0 f117q.sanityn50 -> 0 f117q.sanityn51 -> 0 f117q.sanityn52 -> 1 f117q.sanityn53 -> 1 f117q.sanityn54 -> 1 f117q.sanityn55 -> 0 f117q.sanityn56 -> 1 f117q.sanityn57 -> 1 f117q.sanityn58 -> 1 f117q.sanityn59 -> 0 f117q.sanityn60 -> 1 f117q.sanityn61 -> 1 f117q.sanityn62 -> 1 f117q.sanityn63 -> 1 f117q.sanityn64 -> 1 f117q.sanityn65 -> 0 f117q.sanityn66 -> 1 f117q.sanityn67 -> 1 f117q.sanityn68 -> 0 f117q.sanityn69 -> 0 f117q.sanityn70 -> 0 f117q.sanityn71 -> 0 f117q.sanityn72 -> 0 f117q.sanityn73 -> 0 f117q.sanityn74 -> 0 f117q.sanityn75 -> 0 f117q.sanityn76 -> 1 f117q.sanityn77 -> 0 f117q.sanityn78 -> 1 f117q.sanityn79 -> 0 f117q.sanityn80 -> 1 f117q.sanityn81 -> 1 f117q.sanityn82 -> 0 f117q.sanityn83 -> 0 f117q.sanityn84 -> 1 f117q.sanityn85 -> 0 f117q.sanityn86 -> 1 f117q.sanityn87 -> 1 f117q.sanityn88 -> 1 f117q.sanityn89 -> 0 f117q.sanityn90 -> 1 f117q.sanityn91 -> 0 f117q.sanityn92 -> 0 f117q.sanityn93 -> 1 f117q.sanityn94 -> 0 f117q.sanityn95 -> 1 f117q.sanityn96 -> 0 f117q.sanityn97 -> 1 f117q.sanityn98 -> 0 f117q.sanityn99 -> 0 - unlinked 0 (time 1761282670 ; total 0 ; last 0) total: 100 unlinks in 0 seconds: inf unlinks/second f117q.sanityn0 f117q.sanityn1 f117q.sanityn10 f117q.sanityn11 f117q.sanityn12 f117q.sanityn13 f117q.sanityn14 f117q.sanityn15 f117q.sanityn16 f117q.sanityn17 f117q.sanityn18 f117q.sanityn19 f117q.sanityn2 f117q.sanityn20 f117q.sanityn21 f117q.sanityn22 f117q.sanityn23 f117q.sanityn24 f117q.sanityn25 f117q.sanityn26 f117q.sanityn27 f117q.sanityn28 f117q.sanityn29 f117q.sanityn3 f117q.sanityn30 f117q.sanityn31 f117q.sanityn32 f117q.sanityn33 f117q.sanityn34 f117q.sanityn35 f117q.sanityn36 f117q.sanityn37 f117q.sanityn38 f117q.sanityn39 f117q.sanityn4 f117q.sanityn40 f117q.sanityn41 f117q.sanityn42 f117q.sanityn43 f117q.sanityn44 f117q.sanityn45 f117q.sanityn46 f117q.sanityn47 f117q.sanityn48 f117q.sanityn49 f117q.sanityn5 f117q.sanityn50 f117q.sanityn51 f117q.sanityn52 f117q.sanityn53 f117q.sanityn54 f117q.sanityn55 f117q.sanityn56 f117q.sanityn57 f117q.sanityn58 f117q.sanityn59 f117q.sanityn6 f117q.sanityn60 f117q.sanityn61 f117q.sanityn62 f117q.sanityn63 f117q.sanityn64 f117q.sanityn65 f117q.sanityn66 f117q.sanityn67 f117q.sanityn68 f117q.sanityn69 f117q.sanityn7 f117q.sanityn70 f117q.sanityn71 f117q.sanityn72 f117q.sanityn73 f117q.sanityn74 f117q.sanityn75 f117q.sanityn76 f117q.sanityn77 f117q.sanityn78 f117q.sanityn79 f117q.sanityn8 f117q.sanityn80 f117q.sanityn81 f117q.sanityn82 f117q.sanityn83 f117q.sanityn84 f117q.sanityn85 f117q.sanityn86 f117q.sanityn87 f117q.sanityn88 f117q.sanityn89 f117q.sanityn9 f117q.sanityn90 f117q.sanityn91 f117q.sanityn92 f117q.sanityn93 f117q.sanityn94 f117q.sanityn95 f117q.sanityn96 f117q.sanityn97 f117q.sanityn98 f117q.sanityn99 f117q.sanityn10 f117q.sanityn11 f117q.sanityn12 f117q.sanityn13 f117q.sanityn14 f117q.sanityn15 f117q.sanityn16 f117q.sanityn17 f117q.sanityn18 f117q.sanityn19 f117q.sanityn20 f117q.sanityn21 f117q.sanityn22 f117q.sanityn23 f117q.sanityn24 f117q.sanityn25 f117q.sanityn26 f117q.sanityn27 f117q.sanityn28 f117q.sanityn29 f117q.sanityn30 f117q.sanityn31 f117q.sanityn32 f117q.sanityn33 f117q.sanityn34 f117q.sanityn35 f117q.sanityn36 f117q.sanityn37 f117q.sanityn38 f117q.sanityn39 f117q.sanityn40 f117q.sanityn41 f117q.sanityn42 f117q.sanityn43 f117q.sanityn44 f117q.sanityn45 f117q.sanityn46 f117q.sanityn47 f117q.sanityn48 f117q.sanityn49 f117q.sanityn50 f117q.sanityn51 f117q.sanityn52 f117q.sanityn53 f117q.sanityn54 f117q.sanityn55 f117q.sanityn56 f117q.sanityn57 f117q.sanityn58 f117q.sanityn59 f117q.sanityn60 f117q.sanityn61 f117q.sanityn62 f117q.sanityn63 f117q.sanityn64 f117q.sanityn65 f117q.sanityn66 f117q.sanityn67 f117q.sanityn68 f117q.sanityn69 f117q.sanityn70 f117q.sanityn71 f117q.sanityn72 f117q.sanityn73 f117q.sanityn74 f117q.sanityn75 f117q.sanityn76 f117q.sanityn77 f117q.sanityn78 f117q.sanityn79 f117q.sanityn80 f117q.sanityn81 f117q.sanityn82 f117q.sanityn83 f117q.sanityn84 f117q.sanityn85 f117q.sanityn86 f117q.sanityn87 f117q.sanityn88 f117q.sanityn89 f117q.sanityn90 f117q.sanityn91 f117q.sanityn92 f117q.sanityn93 f117q.sanityn94 f117q.sanityn95 f117q.sanityn96 f117q.sanityn97 f117q.sanityn98 f117q.sanityn99 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117q (15s) == sanityn test 117r: Verify the deleting a striped directory on Trash Can ========================================================== 01:11:19 (1761282679) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain #0 -> 0x200000bd0:0x9bd:0x0 #1 -> 0x240000404:0x989:0x0 Creating 50 files under striped directory: /mnt/lustre/d117r.sanityn/striped_dir total: 50 open/close in 0.29 seconds: 172.33 ops/second f117r.sanityn0 -> 0 f117r.sanityn1 -> 0 f117r.sanityn2 -> 0 f117r.sanityn3 -> 0 f117r.sanityn4 -> 0 f117r.sanityn5 -> 0 f117r.sanityn6 -> 0 f117r.sanityn7 -> 0 f117r.sanityn8 -> 0 f117r.sanityn9 -> 0 f117r.sanityn10 -> 1 f117r.sanityn11 -> 1 f117r.sanityn12 -> 1 f117r.sanityn13 -> 1 f117r.sanityn14 -> 1 f117r.sanityn15 -> 0 f117r.sanityn16 -> 1 f117r.sanityn17 -> 1 f117r.sanityn18 -> 0 f117r.sanityn19 -> 1 f117r.sanityn20 -> 0 f117r.sanityn21 -> 0 f117r.sanityn22 -> 0 f117r.sanityn23 -> 0 f117r.sanityn24 -> 0 f117r.sanityn25 -> 1 f117r.sanityn26 -> 0 f117r.sanityn27 -> 1 f117r.sanityn28 -> 1 f117r.sanityn29 -> 1 f117r.sanityn30 -> 1 f117r.sanityn31 -> 0 f117r.sanityn32 -> 0 f117r.sanityn33 -> 0 f117r.sanityn34 -> 1 f117r.sanityn35 -> 0 f117r.sanityn36 -> 0 f117r.sanityn37 -> 0 f117r.sanityn38 -> 0 f117r.sanityn39 -> 1 f117r.sanityn40 -> 0 f117r.sanityn41 -> 1 f117r.sanityn42 -> 0 f117r.sanityn43 -> 1 f117r.sanityn44 -> 0 f117r.sanityn45 -> 1 f117r.sanityn46 -> 0 f117r.sanityn47 -> 0 f117r.sanityn48 -> 1 f117r.sanityn49 -> 0 - unlinked 0 (time 1761282684 ; total 0 ; last 0) total: 50 unlinks in 0 seconds: inf unlinks/second Check files in Trash Can: f117r.sanityn0 f117r.sanityn1 f117r.sanityn10 f117r.sanityn11 f117r.sanityn12 f117r.sanityn13 f117r.sanityn14 f117r.sanityn15 f117r.sanityn16 f117r.sanityn17 f117r.sanityn18 f117r.sanityn19 f117r.sanityn2 f117r.sanityn20 f117r.sanityn21 f117r.sanityn22 f117r.sanityn23 f117r.sanityn24 f117r.sanityn25 f117r.sanityn26 f117r.sanityn27 f117r.sanityn28 f117r.sanityn29 f117r.sanityn3 f117r.sanityn30 f117r.sanityn31 f117r.sanityn32 f117r.sanityn33 f117r.sanityn34 f117r.sanityn35 f117r.sanityn36 f117r.sanityn37 f117r.sanityn38 f117r.sanityn39 f117r.sanityn4 f117r.sanityn40 f117r.sanityn41 f117r.sanityn42 f117r.sanityn43 f117r.sanityn44 f117r.sanityn45 f117r.sanityn46 f117r.sanityn47 f117r.sanityn48 f117r.sanityn49 f117r.sanityn5 f117r.sanityn6 f117r.sanityn7 f117r.sanityn8 f117r.sanityn9 f117r.sanityn10 f117r.sanityn11 f117r.sanityn12 f117r.sanityn13 f117r.sanityn14 f117r.sanityn15 f117r.sanityn16 f117r.sanityn17 f117r.sanityn18 f117r.sanityn19 f117r.sanityn20 f117r.sanityn21 f117r.sanityn22 f117r.sanityn23 f117r.sanityn24 f117r.sanityn25 f117r.sanityn26 f117r.sanityn27 f117r.sanityn28 f117r.sanityn29 f117r.sanityn30 f117r.sanityn31 f117r.sanityn32 f117r.sanityn33 f117r.sanityn34 f117r.sanityn35 f117r.sanityn36 f117r.sanityn37 f117r.sanityn38 f117r.sanityn39 f117r.sanityn40 f117r.sanityn41 f117r.sanityn42 f117r.sanityn43 f117r.sanityn44 f117r.sanityn45 f117r.sanityn46 f117r.sanityn47 f117r.sanityn48 f117r.sanityn49 FID for /mnt/lustre/d117r.sanityn/striped_dir [0x200000bd1:0xc71:0x0] pfid[0]=0x200000bd0:0x9bd:0x0 [0x200000400:0x2:0x0] [0x200000bd0:0x9be:0x0] pfid[1]=0x240000404:0x989:0x0 [0x240000400:0x2:0x0] [0x240000400:0x10f8:0x0] Obtain LMVEA for virtual dir /mnt/lustre/d117r.sanityn/striped_dir/.Trash: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000bd0:0x9be:0x0] 1 [0x240000400:0x10f8:0x0] #0 -> name: 0x200000bd0:0x9bd:0x0 fid: 0x200000bd0:0x9be:0x0 #1 -> name: 0x240000404:0x989:0x0 fid: 0x240000400:0x10f8:0x0 /mnt/lustre/.lustre/trash: MDT0000 MDT0001 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd0:0x9bb:0x0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd0:0x9bb:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc37:0x0: /mnt/lustre/.lustre/trash/MDT0000/500: /mnt/lustre/.lustre/trash/MDT0001: 0x240000404:0x988:0x0 0x240000406:0x3e:0x0 0x240000406:0xc:0x0 /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x988:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000406:0x3e:0x0: striped_dir /mnt/lustre/.lustre/trash/MDT0001/0x240000406:0x3e:0x0/striped_dir: f117r.sanityn0 f117r.sanityn1 f117r.sanityn10 f117r.sanityn11 f117r.sanityn12 f117r.sanityn13 f117r.sanityn14 f117r.sanityn15 f117r.sanityn16 f117r.sanityn17 f117r.sanityn18 f117r.sanityn19 f117r.sanityn2 f117r.sanityn20 f117r.sanityn21 f117r.sanityn22 f117r.sanityn23 f117r.sanityn24 f117r.sanityn25 f117r.sanityn26 f117r.sanityn27 f117r.sanityn28 f117r.sanityn29 f117r.sanityn3 f117r.sanityn30 f117r.sanityn31 f117r.sanityn32 f117r.sanityn33 f117r.sanityn34 f117r.sanityn35 f117r.sanityn36 f117r.sanityn37 f117r.sanityn38 f117r.sanityn39 f117r.sanityn4 f117r.sanityn40 f117r.sanityn41 f117r.sanityn42 f117r.sanityn43 f117r.sanityn44 f117r.sanityn45 f117r.sanityn46 f117r.sanityn47 f117r.sanityn48 f117r.sanityn49 f117r.sanityn5 f117r.sanityn6 f117r.sanityn7 f117r.sanityn8 f117r.sanityn9 file.1 file.2 /mnt/lustre/.lustre/trash/MDT0001/0x240000406:0xc:0x0: Obtain LMVEA for striped dir /mnt/lustre/d117r.sanityn/striped_dir: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000bd0:0x9be:0x0] 1 [0x240000400:0x10f8:0x0] [0x200000bd0:0x9c0:0x0] #0 -> name: 0x200000bd0:0x9bd:0x0 fid: 0x200000bd0:0x9be:0x0 #1 -> name: 0x240000404:0x989:0x0 fid: 0x240000400:0x10f8:0x0 lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000bd0:0x9be:0x0] 1 [0x240000400:0x10f8:0x0] f117r.sanityn0 f117r.sanityn1 f117r.sanityn10 f117r.sanityn11 f117r.sanityn12 f117r.sanityn13 f117r.sanityn14 f117r.sanityn15 f117r.sanityn16 f117r.sanityn17 f117r.sanityn18 f117r.sanityn19 f117r.sanityn2 f117r.sanityn20 f117r.sanityn21 f117r.sanityn22 f117r.sanityn23 f117r.sanityn24 f117r.sanityn25 f117r.sanityn26 f117r.sanityn27 f117r.sanityn28 f117r.sanityn29 f117r.sanityn3 f117r.sanityn30 f117r.sanityn31 f117r.sanityn32 f117r.sanityn33 f117r.sanityn34 f117r.sanityn35 f117r.sanityn36 f117r.sanityn37 f117r.sanityn38 f117r.sanityn39 f117r.sanityn4 f117r.sanityn40 f117r.sanityn41 f117r.sanityn42 f117r.sanityn43 f117r.sanityn44 f117r.sanityn45 f117r.sanityn46 f117r.sanityn47 f117r.sanityn48 f117r.sanityn49 f117r.sanityn5 f117r.sanityn6 f117r.sanityn7 f117r.sanityn8 f117r.sanityn9 file.1 file.2 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn0,flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn1,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn10,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn11,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn12,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn13,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn14,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn15,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn16,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn17,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn18,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn19,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn2,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn20,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn21,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn22,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn23,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn24,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn25,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn26,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn27,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn28,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn29,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn3,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn30,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn31,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn32,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn33,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn34,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn35,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn36,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn37,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn38,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn39,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn4,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn40,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn41,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn42,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn43,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn44,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn45,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn46,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn47,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn48,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn49,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn5,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn6,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn7,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn8,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn9,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282684 file: /mnt/lustre/d117r.sanityn/striped_dir/file.1,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282683 file: /mnt/lustre/d117r.sanityn/striped_dir/file.2,flags: 0X7EFF00080000, uid: 0 gid: 0 projid: 0 timestamp: 1761282683 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117r (13s) == sanityn test 117s: Create trash shard if it does not exist ========================================================== 01:11:32 (1761282692) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain /mnt/lustre/.lustre/trash: MDT0000 MDT0001 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd0:0x9bb:0x0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd0:0x9bb:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd1:0xc37:0x0: /mnt/lustre/.lustre/trash/MDT0000/500: /mnt/lustre/.lustre/trash/MDT0001: 0x240000404:0x988:0x0 0x240000404:0x98a:0x0 0x240000406:0x3e:0x0 0x240000406:0xc:0x0 /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x988:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x98a:0x0: file.1 /mnt/lustre/.lustre/trash/MDT0001/0x240000406:0x3e:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000406:0xc:0x0: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000bd0:0x9c3:0x0] 1 [0x240000404:0x98a:0x0] mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117s (8s) == sanityn test 117t: UID/GID/ProjID for files in Trash Can ========================================================== 01:11:40 (1761282700) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain file: /mnt/lustre/d117t.sanityn/.Trash/tfile1,flags: 0X80002 unrm, uid: 0 gid: 0 projid: 0 timestamp: 1761282704 File: /mnt/lustre/d117t.sanityn/.Trash/tfile1 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826937493 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 01:11:44.000000000 -0400 Modify: 2025-10-24 01:11:44.000000000 -0400 Change: 2025-10-24 01:11:44.000000000 -0400 Birth: 2025-10-24 01:11:44.000000000 -0400 trash_can_uid=4294967294 trash_can_gid=4294967294 trash_can_projid=4294967294 === Checking ugp ID on MDT === oleg128-server: debugfs 1.47.1-wc2 (08-Nov-2024) Inode: 519340 Type: regular Mode: 0644 Flags: 0x80002 Generation: 2145967018 Version: 0x00000002:00019a35 User: -2 Group: -2 Project: -2 Size: 0 File ACL: 0 Links: 1 Blockcount: 0 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x68fb0a90:00000000 -- Fri Oct 24 01:11:44 2025 atime: 0x68fb0a90:00000000 -- Fri Oct 24 01:11:44 2025 mtime: 0x68fb0a90:00000000 -- Fri Oct 24 01:11:44 2025 crtime: 0x68fb0a90:58370200 -- Fri Oct 24 01:11:44 2025 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x200000bd1:0xc95:0x0] compat=0 incompat=0 user.job (5) = "lfs.0" linkea: idx=0 parent=[0x200000bd0:0x9c7:0x0] name='tfile1' trusted.lov (56) = d0 0b d1 0b 01 00 00 00 95 0c 00 00 00 00 00 00 d1 0b 00 00 02 00 00 00 00 00 40 00 01 00 00 00 01 04 00 80 02 00 00 00 a6 1a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 trusted.som (24) = 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 trusted.unrm (24) = 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 90 0a fb 68 00 00 00 00 EXTENTS: oleg128-server: debugfs 1.47.1-wc2 (08-Nov-2024) oleg128-server: debugfs 1.47.1-wc2 (08-Nov-2024) oleg128-server: debugfs 1.47.1-wc2 (08-Nov-2024) === Checking ugp ID on OST === oleg128-server: debugfs 1.47.1-wc2 (08-Nov-2024) Inode: 3525 Type: regular Mode: 0666 Flags: 0x80000 Generation: 2153201545 Version: 0x00000000:00000000 User: -2 Group: -2 Project: -2 Size: 0 File ACL: 0 Links: 1 Blockcount: 0 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x00000000:00000000 -- Wed Dec 31 19:00:00 1969 atime: 0x00000000:00000000 -- Wed Dec 31 19:00:00 1969 mtime: 0x00000000:00000000 -- Wed Dec 31 19:00:00 1969 crtime: 0x68fb0a6d:1982c300 -- Fri Oct 24 01:11:09 2025 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x280000401:0x1aa6:0x0] compat=8 incompat=0 EXTENTS: oleg128-server: debugfs 1.47.1-wc2 (08-Nov-2024) oleg128-server: debugfs 1.47.1-wc2 (08-Nov-2024) oleg128-server: debugfs 1.47.1-wc2 (08-Nov-2024) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d117t.sanityn/tfile2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mkdir] [/mnt/lustre/d117t.sanityn/tdir2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [unlink] [/mnt/lustre/d117t.sanityn/tfile2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [rmdir] [/mnt/lustre/d117t.sanityn/tdir2] file: /mnt/lustre/d117t.sanityn/.Trash/tfile2,flags: 0X80002 unrm, uid: 500 gid: 500 projid: 0 timestamp: 1761282710 File: /mnt/lustre/d117t.sanityn/.Trash/tfile2 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826937494 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 500/sanityusr) Gid: ( 500/sanityusr) Access: 2025-10-24 01:11:50.000000000 -0400 Modify: 2025-10-24 01:11:50.000000000 -0400 Change: 2025-10-24 01:11:50.000000000 -0400 Birth: 2025-10-24 01:11:50.000000000 -0400 using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg128-client.virtnet /mnt/lustre (opts:) Stopping client oleg128-client.virtnet /mnt/lustre opts: Stopping clients: oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg128-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg128-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg128-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg128-server tune2fs 1.47.1-wc2 (08-Nov-2024) tune2fs 1.47.1-wc2 (08-Nov-2024) tune2fs 1.47.1-wc2 (08-Nov-2024) tune2fs 1.47.1-wc2 (08-Nov-2024) sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg128-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.28,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg128-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.28,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg128-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.28,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/56360/testresults/sanityn-ldiskfs-DNE-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.28,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg128-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg128-server: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg128-server: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg128-server: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg128-server: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Starting client oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre Started clients oleg128-client.virtnet: 192.168.201.128@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 Starting client oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre2 Started clients oleg128-client.virtnet: 192.168.201.128@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff931f065b3800.idle_timeout=debug osc.lustre-OST0000-osc-ffff931f0741f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff931f065b3800.idle_timeout=debug osc.lustre-OST0001-osc-ffff931f0741f000.idle_timeout=debug disable quota as required mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117t (134s) == sanityn test 200: service remains healthy while able to process request ========================================================== 01:13:54 (1761282834) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=3 striped dir -i0 -c2 -H crush2 /mnt/lustre/d200.sanityn fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0857 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 52.8024 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 84.5211 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 116.265 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 148.005 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 180.768 s, 0.0 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg128-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff931f065b3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff931f065b3800.ost_server_uuid in FULL state after 0 sec PASS 200 (187s) cleanup: ====================================================== == sanityn test complete, duration 7528 sec ============== 01:17:02 (1761283022) === sanityn: start cleanup 01:17:03 (1761283023) === Stopping clients: oleg128-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg128-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 01:18:29 (1761283109) ===