-----============= acceptance-small: sanityn ============----- Wed Jul 24 23:55:40 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 27 28 skipping tests SLOW=no: 33a === sanityn: start setup 23:55:45 (1721879745) === Starting client oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 Started clients oleg154-client.virtnet: 192.168.201.154@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) oleg154-client.virtnet: executing check_config_client /mnt/lustre oleg154-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg154-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800abf93800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8800b5882800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800abf93800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5882800.idle_timeout=debug disable quota as required oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanityn: finish setup 23:55:54 (1721879754) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114428 s, 91.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7561] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 23:55:54 (1721879754) PASS 0 (3s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 23:55:57 (1721879757) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (2s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 23:55:59 (1721879759) -rw-r--r-- 1 root root 0 Jul 24 23:55 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (2s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 23:56:01 (1721879761) -rw-r--r-- 1 root root 0 Jul 24 23:56 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (2s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 23:56:03 (1721879763) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (1s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 23:56:04 (1721879764) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (2s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 23:56:06 (1721879766) total 0 -rwxrwxrwx 1 root root 0 Jul 24 23:55 f2a -rwxrwxrwx 1 root root 0 Jul 24 23:56 f2b total 0 -rwxrwxrwx 1 root root 0 Jul 24 23:55 f2a -rwxrwxrwx 1 root root 0 Jul 24 23:56 f2b running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.0055548 s, 92.2 kB/s PASS 2e (2s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 23:56:08 (1721879768) /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has type file OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has perms 0777 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by user #500 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by group #500 OK Can't lstat /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn: No such file or directory PASS 2f (2s) == sanityn test 2g: check blocks update on sync write ==== 23:56:10 (1721879770) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0722209 s, 29.0 MB/s /mnt/lustre/f2g.sanityn has 4096 blocks /mnt/lustre2/f2g.sanityn has 4096 blocks PASS 2g (1s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 23:56:12 (1721879772) PASS 3 (2s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 23:56:13 (1721879773) PASS 4 (3s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 23:56:17 (1721879777) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (1s) == sanityn test 6: remove of open file on other node ============================================================================ 23:56:18 (1721879778) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (2s) == sanityn test 7: remove of open directory on other node ======================================================================= 23:56:20 (1721879780) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (2s) == sanityn test 8: remove of open special file on other node ==================================================================== 23:56:22 (1721879782) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (2s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 23:56:24 (1721879784) PASS 9a (2s) == sanityn test 9b: append to striped sparse file ======== 23:56:26 (1721879786) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0157616 s, 66.5 MB/s 3+0 records in 3+0 records out 3 bytes (3 B) copied, 0.0098867 s, 0.3 kB/s Data read (expecting 'foo'): foo PASS 9b (2s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 23:56:28 (1721879788) 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00229809 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00267129 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00366176 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00456736 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00361658 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.0038295 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.0031399 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00366 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00441443 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00302775 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00321726 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00404584 s, 0.2 kB/s PASS 10a (2s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 23:56:30 (1721879790) 1+0 records in 1+0 records out 3072 bytes (3.1 kB) copied, 0.00385325 s, 797 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0116098 s, 353 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB) copied, 0.000246713 s, 12.5 MB/s PASS 10b (1s) == sanityn test 11: execution of file opened for write should return error ============================================================== 23:56:31 (1721879791) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d11 multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7561 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 301: /mnt/lustre2/d11/f: Text file busy PASS 11 (2s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 23:56:33 (1721879793) warning: '-runtime' deprecated, use '-t runtime' instead - link 1993 (time 1721879804.07 total 10.00 last 199.28) - link 4419 (time 1721879814.07 total 20.00 last 242.56) - link 6718 (time 1721879824.08 total 30.00 last 229.87) - link 8765 (time 1721879834.08 total 40.01 last 204.67) - link 10000 (time 1721879840.16 total 46.09 last 203.12) - link 12056 (time 1721879850.16 total 56.09 last 205.55) - link 13953 (time 1721879860.16 total 66.09 last 189.65) - link 16046 (time 1721879870.16 total 76.09 last 209.24) - link 18101 (time 1721879880.17 total 86.09 last 205.48) - link 20000 (time 1721879889.89 total 95.82 last 195.29) total: 20890 link in 100.00 seconds: 208.90 ops/second using seed 398891221 running for 100 seconds - stat 10000 (time 1721879798 ; total 3 ; last 3) - stat 20000 (time 1721879801 ; total 6 ; last 3) - stat 30000 (time 1721879804 ; total 9 ; last 3) - stat 40000 (time 1721879807 ; total 12 ; last 3) - stat 50000 (time 1721879810 ; total 15 ; last 3) - stat 60000 (time 1721879813 ; total 18 ; last 3) - stat 70000 (time 1721879816 ; total 21 ; last 3) - stat 80000 (time 1721879819 ; total 24 ; last 3) - stat 90000 (time 1721879822 ; total 27 ; last 3) - stat 100000 (time 1721879825 ; total 30 ; last 3) - stat 110000 (time 1721879828 ; total 33 ; last 3) - stat 120000 (time 1721879831 ; total 36 ; last 3) - stat 130000 (time 1721879834 ; total 39 ; last 3) - stat 140000 (time 1721879837 ; total 42 ; last 3) - stat 150000 (time 1721879841 ; total 46 ; last 4) - stat 160000 (time 1721879844 ; total 49 ; last 3) - stat 170000 (time 1721879851 ; total 56 ; last 7) - stat 179966 (time 1721879862 ; total 67 ; last 11) - stat 180000 (time 1721879862 ; total 67 ; last 0) - stat 190000 (time 1721879871 ; total 76 ; last 9) - stat 200000 (time 1721879881 ; total 86 ; last 10) - stat 210000 (time 1721879891 ; total 96 ; last 10) total: 215246 stats in 100 seconds: 2152.459961 stats/second - unlinked 0 (time 1721879845 ; total 0 ; last 0) - unlinked 10000 (time 1721879905 ; total 60 ; last 60) - unlinked 20000 (time 1721879933 ; total 88 ; last 28) unlink(/mnt/lustre2/lockdir/lockfile20890) error: No such file or directory total: 20889 unlinks in 90 seconds: 232.100006 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (22558) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (22561) - No such process PASS 12 (143s) == sanityn test 13: test directory page revocation ======= 23:58:56 (1721879936) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d13 f13 f13 PASS 13 (2s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 23:58:58 (1721879938) striped dir -i0 -c2 -H all_char /mnt/lustre/d14aa.sanityn multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7561 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 336: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4759: 23964 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (1s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 23:58:59 (1721879939) striped dir -i0 -c2 -H all_char /mnt/lustre/d14ab.sanityn open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4759: 24606 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (2s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 23:59:01 (1721879941) striped dir -i0 -c2 -H crush2 /mnt/lustre/d14b.sanityn truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 351: 25247 Terminated $DIR1/$tdir/sleep 60 PASS 14b (1s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 23:59:02 (1721879942) striped dir -i0 -c2 -H crush2 /mnt/lustre/d14c.sanityn cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 363: 25889 Terminated $DIR1/$tdir/sleep 60 PASS 14c (1s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 23:59:03 (1721879943) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d14d.sanityn chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 374: 26532 Terminated $DIR1/$tdir/sleep 60 PASS 14d (2s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 23:59:05 (1721879945) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_CODE=117440512 CLIENT_OS_VERSION_ID=7 STRIPECOUNT=2 ORIGFREE=7207372 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7207372kB free gt MAXFREE 800000kB, increase 800000 (or reduce test fs size) to proceed Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 SKIP 15 (6s) == sanityn test 16a: 2500 iterations of dual-mount fsx === 23:59:11 (1721879951) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.152913 s, 54.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.143417 s, 58.5 MB/s Chance of close/open is 1 in 50 Seed set to 9952 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fallocating to largest ever: 0x7f05b5 fallocating to largest ever: 0x991540 fallocating to largest ever: 0x9ab962 truncating to largest ever: 0x9f77a7 000100[0] 1721879953.938663 mapread 0x1b6e42 thru 0x1be91b (0x7ada bytes) 000200[0] 1721879954.683237 trunc from 0x926454 to 0x3f76b0 (0x52eda3 bytes) 000300[1] 1721879955.569482 mapwrite 0x9a25a4 thru 0x9b08ba (0xe317 bytes) 000400[1] 1721879956.420100 fallocate 0x61a433 thru 0x620ecc (0x6a9a bytes) 000500[0] 1721879957.341882 write 0x6ac8b7 thru 0x6aea96 (0x21e0 bytes) 000700[0] 1721879958.989383 punch from 0x02ea50 to 0x03e22a (0xf7db bytes) fallocating to largest ever: 0xa00000 000800[1] 1721879959.760866 punch from 0x2cdd33 to 0x2ddaf8 (0xfdc6 bytes) 000900[1] 1721879960.599227 mapwrite 0x7bd035 thru 0x7c3d8e (0x6d5a bytes) 001000[0] 1721879961.468106 mapread 0x0452cf thru 0x0532cb (0xdffd bytes) 001100[1] 1721879962.271606 write 0x2281f7 thru 0x22f7d6 (0x75e0 bytes) 001300[1] 1721879963.970523 mapwrite 0x56421d thru 0x572872 (0xe656 bytes) 001500[1] 1721879965.528638 fallocate 0x7bfa5c thru 0x7c0839 (0xdde bytes) 001600[0] 1721879966.526577 punch from 0x202d15 to 0x20bcc8 (0x8fb4 bytes) 001700[1] 1721879967.412082 trunc from 0x75cbd6 to 0x6221e8 (0x13a9ed bytes) 001900[0] 1721879969.173956 fallocate 0x5cb96e thru 0x5cc3ab (0xa3e bytes) 002000[0] 1721879970.078907 trunc from 0x9eefdf to 0x4f6800 (0x4f87de bytes) 002100[0] 1721879970.966723 trunc from 0x960a2d to 0x70cd99 (0x253c93 bytes) 002500[1] 1721879974.318556 fallocate 0x2da5cb thru 0x2e724d (0xcc83 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 9974 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fallocating to largest ever: 0x1fd800 truncating to largest ever: 0x6a3533 truncating to largest ever: 0x83e687 truncating to largest ever: 0x9c9fa2 000200[1] 1721879975.712131 mapread 0x22e000 thru 0x23ccb9 (0xecba bytes) truncating to largest ever: 0x9f5c34 000300[1] 1721879976.441365 mapwrite 0x53b000 thru 0x547537 (0xc538 bytes) truncating to largest ever: 0x9f801e 000400[0] 1721879977.251657 fallocate 0x7ee1e9 thru 0x7f58e7 (0x76ff bytes) 000500[0] 1721879977.765261 trunc from 0x57c615 to 0x7393a4 (0x1bcd90 bytes) fallocating to largest ever: 0x9ff210 000700[1] 1721879978.969480 fallocate 0x1b79cb thru 0x1c34b0 (0xbae6 bytes) 000800[0] 1721879979.781650 trunc from 0x953497 to 0x2a1ad2 (0x6b19c4 bytes) 001000[1] 1721879981.304192 mapread 0x15d000 thru 0x15ed2d (0x1d2e bytes) 001100[1] 1721879981.984988 fallocate 0x990927 thru 0x9993bf (0x8a99 bytes) 001200[0] 1721879982.667825 mapread 0x007000 thru 0x0115fd (0xa5fe bytes) fallocating to largest ever: 0xa00000 001500[1] 1721879984.451423 trunc from 0x991000 to 0x54d7e5 (0x44381a bytes) 001700[0] 1721879985.805847 mapwrite 0x87a000 thru 0x87a502 (0x503 bytes) 001800[0] 1721879986.475586 mapread 0x106000 thru 0x10d19a (0x719b bytes) 001900[0] 1721879987.344829 write 0x736000 thru 0x7375e6 (0x15e7 bytes) 002000[0] 1721879988.231615 trunc from 0x0c274c to 0x9fcae6 (0x93a39b bytes) 002100[0] 1721879989.119218 trunc from 0x268982 to 0x92c801 (0x6c3e80 bytes) 002300[1] 1721879990.724604 write 0x2bf000 thru 0x2cc409 (0xd40a bytes) 002400[0] 1721879991.602415 write 0x1ce000 thru 0x1d953a (0xb53b bytes) All operations completed A-OK! PASS 16a (42s) == sanityn test 16b: 2500 iterations of dual-mount fsx at small size ========================================================== 23:59:53 (1721879993) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.14175 s, 59.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.136644 s, 61.4 MB/s Chance of close/open is 1 in 50 Seed set to 9994 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x727697 truncating to largest ever: 0x932a27 fallocating to largest ever: 0x9dc8bc fallocating to largest ever: 0x9f349b 000200[0] 1721879996.049840 trunc from 0x10d84f to 0x825f52 (0x718704 bytes) 000300[1] 1721879996.653734 fallocate 0x9e257a thru 0x9e3f04 (0x198b bytes) 000500[1] 1721879998.025272 write 0x0c7e23 thru 0x0c8671 (0x84f bytes) 000700[0] 1721879999.583446 punch from 0x34aaf4 to 0x34adc1 (0x2ce bytes) 000800[0] 1721880000.299493 write 0x61df47 thru 0x61ea82 (0xb3c bytes) 000900[1] 1721880001.048644 trunc from 0x99d78b to 0x311cd9 (0x68bab1 bytes) 001000[1] 1721880001.817361 mapread 0x130d6a thru 0x131536 (0x7cd bytes) truncating to largest ever: 0x9fe0bf 001300[1] 1721880003.983362 punch from 0x083c74 to 0x084c7d (0x100a bytes) 001400[0] 1721880004.757033 mapwrite 0x0bc97d thru 0x0bdd16 (0x139a bytes) 001500[1] 1721880005.463466 read 0x3cbba2 thru 0x3cda02 (0x1e61 bytes) 001600[1] 1721880006.155464 trunc from 0x995a26 to 0x8d4050 (0xc19d5 bytes) 001800[0] 1721880007.493933 mapread 0x71a05b thru 0x71b5ca (0x1570 bytes) 001900[1] 1721880008.298440 fallocate 0x62224f thru 0x62398c (0x173e bytes) 002000[1] 1721880008.960520 punch from 0x238c2f to 0x239780 (0xb52 bytes) 002100[1] 1721880009.638637 read 0x25affe thru 0x25c8e7 (0x18ea bytes) 002200[0] 1721880010.334727 mapread 0x01b40e thru 0x01c0cf (0xcc2 bytes) 002300[0] 1721880011.030814 mapread 0x4e19ed thru 0x4e34fa (0x1b0e bytes) skipping zero size write 002400[1] 1721880011.698687 write 0x9829a6 thru 0x982cdb (0x336 bytes) All operations completed A-OK! PASS 16b (20s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 00:00:13 (1721880013) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.134978 s, 62.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.140256 s, 59.8 MB/s Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Chance of close/open is 1 in 50 Seed set to 25 fd 0: /mnt/lustre/f16c.sanityn fd 1: /mnt/lustre2/f16c.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read fallocating to largest ever: 0x2621c1 fallocating to largest ever: 0x7fc793 fallocating to largest ever: 0x91b40f truncating to largest ever: 0x94210f truncating to largest ever: 0x9f0237 000200[1] 1721880027.185375 trunc from 0x119cbe to 0x66ab90 (0x550ed3 bytes) 000300[1] 1721880028.141172 fallocate 0x14ab42 thru 0x14cff7 (0x24b6 bytes) 000500[1] 1721880029.951448 trunc from 0x26d808 to 0x1e9704 (0x84103 bytes) 000800[0] 1721880032.766194 mapwrite 0x3795e4 thru 0x37c503 (0x2f20 bytes) 000900[1] 1721880033.847613 mapwrite 0x8d8aca thru 0x8df19c (0x66d3 bytes) 001100[0] 1721880035.644906 mapread 0x2a3b39 thru 0x2ab0b8 (0x7580 bytes) fallocating to largest ever: 0x9fd486 001200[0] 1721880036.605773 trunc from 0x4d0f89 to 0x65159a (0x180612 bytes) 001300[1] 1721880037.522960 punch from 0x2955ca to 0x296b59 (0x1590 bytes) 001400[1] 1721880038.541718 trunc from 0x990355 to 0x03e685 (0x951ccf bytes) 001500[0] 1721880039.693042 write 0x40ea7c thru 0x415dc6 (0x734b bytes) 001700[1] 1721880041.779675 read 0x123a72 thru 0x129eaf (0x643e bytes) 002000[1] 1721880044.749668 punch from 0x29e84c to 0x2ae4f8 (0xfcad bytes) 002100[1] 1721880045.711287 punch from 0x62d8e2 to 0x6311b6 (0x38d5 bytes) 002200[0] 1721880046.578877 trunc from 0x40673a to 0x51af14 (0x1147db bytes) 002300[1] 1721880047.642217 write 0x8f784f thru 0x902681 (0xae33 bytes) 002400[0] 1721880048.859931 trunc from 0x5bf180 to 0x111fd6 (0x4ad1a9 bytes) 002500[0] 1721880049.966724 read 0x3dc3c8 thru 0x3ea754 (0xe38d bytes) All operations completed A-OK! PASS 16c (38s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 00:00:51 (1721880051) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.161001 s, 52.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.159164 s, 52.7 MB/s ldlm.namespaces.MGC192.168.201.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800abf93800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b5882800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800abf93800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b5882800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800abf93800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800abf93800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 1.66323 s, 63.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.551715 s, 190 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.617815 s, 170 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 1.15162 s, 91.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.701783 s, 149 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.786579 s, 133 MB/s PASS 16d (8s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 00:00:59 (1721880059) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.065158 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102181 s, 103 MB/s PASS 16e (2s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 00:01:01 (1721880061) TIMEOUT PASS 16f (21s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 00:01:22 (1721880082) TIMEOUT PASS 16g (22s) == sanityn test 16h: mmap read after truncate file ======= 00:01:44 (1721880104) striped dir -i0 -c2 -H all_char /mnt/lustre/d16h.sanityn ------- before truncate -------- 0003220 3035 3a31 3035 3a31 2f3a 6f68 656d 732f 0003240 6e61 7469 7579 7273 3a31 622f 6e69 622f 0003260 7361 0a68 0003264 ------- after truncate -------- 0001460 7373 6468 783a 373a 3a34 3437 503a 6972 0001500 6976 656c 6567 732d 7065 7261 7461 6465 0001520 5320 4853 2f3a 6176 2f72 0001532 PASS 16h (1s) == sanityn test 16i: read after truncate file ============ 00:01:45 (1721880105) striped dir -i0 -c2 -H crush2 /mnt/lustre/d16i.sanityn ------- before truncate -------- 0003220 3035 3a31 3035 3a31 2f3a 6f68 656d 732f 0003240 6e61 7469 7579 7273 3a31 622f 6e69 622f 0003260 7361 0a68 0003264 ------- after truncate -------- 0001460 7373 6468 783a 373a 3a34 3437 503a 6972 0001500 6976 656c 6567 732d 7065 7261 7461 6465 0001520 5320 4853 2f3a 6176 2f72 0001532 PASS 16i (2s) == sanityn test 16j: race dio with buffered i/o ========== 00:01:47 (1721880107) 1+0 records in 1+0 records out 26214400 bytes (26 MB) copied, 0.355895 s, 73.7 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.271437 s, 96.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.318981 s, 82.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.173477 s, 151 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.216613 s, 121 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.20164 s, 130 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.304808 s, 86.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.281385 s, 93.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.295922 s, 88.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.367102 s, 71.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.445531 s, 58.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.231494 s, 113 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.243853 s, 108 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.270338 s, 97.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.350241 s, 74.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.219944 s, 119 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.286468 s, 91.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.30591 s, 85.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.356913 s, 73.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (5s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 00:01:53 (1721880113) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fallocating to largest ever: 0x3428364 skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fallocating to largest ever: 0x3428364 skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fallocating to largest ever: 0x3428364 skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fallocating to largest ever: 0x3428364 skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fallocating to largest ever: 0x3428364 skipping zero size read truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db fallocating to largest ever: 0x3f1a2be truncating to largest ever: 0x3c711db fallocating to largest ever: 0x3f1a2be fallocating to largest ever: 0x3f1a2be fallocating to largest ever: 0x3f1a2be fallocating to largest ever: 0x3f1a2be fallocating to largest ever: 0x3f8dc08 fallocating to largest ever: 0x3f8dc08 fallocating to largest ever: 0x3f8dc08 fallocating to largest ever: 0x3f8dc08 fallocating to largest ever: 0x3f8dc08 All operations completed A-OK! + fsx 1 done: rc=0 Thu Jul 25 00:02:09 EDT 2024 All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 2 done: rc=0 Thu Jul 25 00:02:10 EDT 2024 + fsx 3 done: rc=0 Thu Jul 25 00:02:10 EDT 2024 + fsx 4 done: rc=0 Thu Jul 25 00:02:10 EDT 2024 + fsx 5 done: rc=0 Thu Jul 25 00:02:10 EDT 2024 PASS 16k (20s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 00:02:12 (1721880132) fail_loc=0x8000030a PASS 17 (4s) == sanityn test 18: mmap sanity check =========================================================================================== 00:02:16 (1721880136) mmap test1: basic mmap operation (PASS, 0.018213s) mmap test2: MAP_PRIVATE not write back (PASS, 0.01341s) mmap test3: concurrent mmap ops on two nodes (PASS, 2.3446s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.0655s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.025901s) mmap test6: check mmap write/read content on two nodes (PASS, 0.028136s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.3473s) mmap test11: multi-thread shared mmap (PASS, 0.32371s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0162s) PASS 18 (10s) == sanityn test 19: test concurrent uncached read races ========================================================================= 00:02:26 (1721880146) 32+0 records in 32+0 records out 16777216 bytes (17 MB) copied, 0.148361 s, 113 MB/s loop 5 loop 10 loop 15 loop 20 PASS 19 (7s) == sanityn test 20: test extra readahead page left in cache ============================================================== 00:02:34 (1721880154) striped dir -i0 -c2 -H all_char /mnt/lustre/d20.sanityn PASS 20 (1s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 00:02:35 (1721880155) striped dir -i1 -c2 -H crush2 /mnt/lustre/d21.sanityn rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (1s) == sanityn test 23: others should see updated atime while another read============================================================== 00:02:36 (1721880156) now is 1721880157 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7561 new atime is 1721880218 PASS 23 (63s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 00:03:39 (1721880219) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 33512 3573508 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5820 3601200 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 39332 7174708 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2060 1285628 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 33512 3573508 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 5820 3601200 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 39332 7174708 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 350 999.7K 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1000.0K 279 999.7K 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 256.0K 375 255.6K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 256.0K 372 255.6K 1% /mnt/lustre[OST:1] filesystem_summary: 511.9K 629 511.3K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 350 999.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1000.0K 279 999.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 256.0K 375 255.6K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 256.0K 372 255.6K 1% /mnt/lustre2[OST:1] filesystem_summary: 511.9K 629 511.3K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 1.3G 2.0M 1.2G 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1.3G 1.8M 1.2G 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3.7G 32.7M 3.4G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.7G 5.7M 3.4G 1% /mnt/lustre[OST:1] filesystem_summary: 7.3G 38.4M 6.8G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 350 1023650 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1024000 279 1023721 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 262144 375 261769 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 262144 372 261772 1% /mnt/lustre2[OST:1] filesystem_summary: 524170 629 523541 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 33512 3573508 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5820 3601200 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 39332 7174708 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 350 999.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1000.0K 279 999.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 256.0K 375 255.6K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 256.0K 372 255.6K 1% /mnt/lustre2[OST:1] filesystem_summary: 511.9K 629 511.3K 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 350 1023650 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1024000 279 1023721 1% /mnt/lustre[MDT:1] filesystem_summary: 2048000 629 2047371 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 350 1023650 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1024000 279 1023721 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 262144 375 261769 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 262144 372 261772 1% /mnt/lustre2[OST:1] filesystem_summary: 524170 629 523541 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 33512 3573508 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5820 3601200 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 39332 7174708 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2060 1285628 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 33512 3573508 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 5820 3601200 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 39332 7174708 1% /mnt/lustre2 PASS 24a (1s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 00:03:41 (1721880221) PASS 24b (2s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 00:03:42 (1721880222) running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (1s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 00:03:43 (1721880223) running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] PASS 25b (2s) == sanityn test 26a: allow mtime to get older ============ 00:03:45 (1721880225) utime: good mknod times 1721880224 <= 1721880225 <= 1721880225 for /mnt/lustre/f26a utime: good mknod times 1721880224 <= 1721880225 <= 1721880225 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (2s) == sanityn test 26b: sync mtime between ost and mds ====== 00:03:47 (1721880227) PASS 26b (4s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 00:03:51 (1721880231) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7561 File: '/mnt/lustre/f26c.sanityn' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279609 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2024-07-25 00:03:51.000000000 -0400 Birth: - File: '/mnt/lustre2/f26c.sanityn' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279609 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2024-07-25 00:03:51.000000000 -0400 Birth: - PASS 26c (2s) SKIP: sanityn test_27 skipping excluded test 27 SKIP: sanityn test_28 skipping ALWAYS excluded test 28 == sanityn test 30: recreate file race =================== 00:03:54 (1721880234) striped dir -i0 -c2 -H all_char /mnt/lustre/d30.sanityn 0 PASS 30 (3s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 00:03:57 (1721880237) striped dir -i1 -c2 -H all_char /mnt/lustre/d31a.sanityn fail_loc=0x314 PASS 31a (3s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 00:04:00 (1721880240) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg154-server mds-ost sync done. Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 striped dir -i1 -c2 -H crush2 /mnt/lustre/d31b.sanityn fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.154@tcp:/lustre 7666232 39332 7174680 1% /mnt/lustre2 PASS 31b (11s) == sanityn test 31r: open-rename(replace) race =========== 00:04:11 (1721880251) -rw-r--r-- 1 root root 0 Jul 25 00:04 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (4s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 00:04:15 (1721880255) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1211: 17003 Terminated tail -f $tfile PASS 31s (2s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 00:04:17 (1721880257) PASS 31t (2s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 00:04:19 (1721880259) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (1s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 00:04:20 (1721880260) Stopping /mnt/lustre-mds1 (opts:) on oleg154-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-MDT0000 create file on 2nd stripe under striped directory /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre/d33c.sanityn/striped touch /mnt/lustre2/d33c.sanityn/striped/subfile SoLC count 2 Below operations shouldn't trigger Solc: create file on 2nd stripe after setattr chmod 777 /mnt/lustre/d33c.sanityn/striped; touch /mnt/lustre2/d33c.sanityn/striped/subfile create remote dir and local dir /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote create local dir after remote dir creation transaction commit /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote.3; do_facet mds2 /home/green/git/lustre-release/lustre/utils/lctl set_param -n osd*.*MDT0001.force_sync 1; mkdir /mnt/lustre/d33c.sanityn/local.3 Solc count 0 PASS 33c (14s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 00:04:34 (1721880274) remote directory create /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre2/d33d.sanityn/subdir CoS count 2 cross-MDT rename mkdir /mnt/lustre/d33d.sanityn; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn/d1; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33d.sanityn/d2; touch /mnt/lustre/d33d.sanityn/d1/src mv /mnt/lustre2/d33d.sanityn/d1/src /mnt/lustre2/d33d.sanityn/d2/tgt CoS count 2 migrate /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs migrate -m 1 /mnt/lustre2/d33d.sanityn CoS count 2 PASS 33d (11s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 00:04:45 (1721880285) plain dir creation mkdir /mnt/lustre2/d33e.sanityn/plain open file and write echo abc > /mnt/lustre2/d33e.sanityn/f33e.sanityn append write echo dfg >> /mnt/lustre2/d33e.sanityn/f33e.sanityn setattr touch /mnt/lustre2/d33e.sanityn/f33e.sanityn file unlink rm /mnt/lustre2/d33e.sanityn/f33e.sanityn plain dir unlink rmdir /mnt/lustre2/d33e.sanityn/plain striped directory creation /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre2/d33e.sanityn/striped set default LMV to create striped subdir /home/green/git/lustre-release/lustre/utils/lfs setdirstripe -D -c 2 /mnt/lustre/d33e.sanityn striped subdir creation createmany -d /mnt/lustre/d33e.sanityn/subdir 100 total: 100 mkdir in 0.22 seconds: 449.19 ops/second sub file creation and write createmany -o /mnt/lustre/d33e.sanityn/subfile 100; echo abc > /mnt/lustre/d33e.sanityn/subfile1 total: 100 open/close in 0.21 seconds: 466.45 ops/second sub file append write echo dfg >> /mnt/lustre2/d33e.sanityn/subfile2 subdir setatttr touch /mnt/lustre2/d33e.sanityn/subdir1 subdir unlink unlinkmany -d /mnt/lustre/d33e.sanityn/subdir 100 - unlinked 0 (time 1721880286 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second sub file unlink unlinkmany /mnt/lustre2/d33e.sanityn/subfile 100 - unlinked 0 (time 1721880287 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second sub file creation follows striped dir chmod chmod 777 /mnt/lustre/d33e.sanityn/striped; touch /mnt/lustre/d33e.sanityn/striped/subfile striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn/striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn CoS count 0 PASS 33e (4s) == sanityn test 34: no lock timeout under IO ============= 00:04:49 (1721880289) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg154-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800abf93800.ost_server_uuid,osc.lustre-OST0000-osc-ffff8800b5882800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800abf93800.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0000-osc-ffff8800b5882800.ost_server_uuid in FULL state after 0 sec oleg154-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800abf93800.ost_server_uuid,osc.lustre-OST0001-osc-ffff8800b5882800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800abf93800.ost_server_uuid in IDLE state after 0 sec osc.lustre-OST0001-osc-ffff8800b5882800.ost_server_uuid in IDLE state after 0 sec oleg154-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800abf93800.ost_server_uuid,osc.lustre-OST0000-osc-ffff8800b5882800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800abf93800.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0000-osc-ffff8800b5882800.ost_server_uuid in FULL state after 0 sec oleg154-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800abf93800.ost_server_uuid,osc.lustre-OST0001-osc-ffff8800b5882800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800abf93800.ost_server_uuid in IDLE state after 0 sec osc.lustre-OST0001-osc-ffff8800b5882800.ost_server_uuid in IDLE state after 0 sec oleg154-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800abf93800.ost_server_uuid,osc.lustre-OST0000-osc-ffff8800b5882800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800abf93800.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0000-osc-ffff8800b5882800.ost_server_uuid in FULL state after 0 sec oleg154-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800abf93800.ost_server_uuid,osc.lustre-OST0001-osc-ffff8800b5882800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800abf93800.ost_server_uuid in IDLE state after 0 sec osc.lustre-OST0001-osc-ffff8800b5882800.ost_server_uuid in IDLE state after 0 sec PASS 34 (82s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 00:06:11 (1721880371) striped dir -i1 -c2 -H crush /mnt/lustre/d35.sanityn fail_loc=0x80000317 Race attempt 0 Wait for 28400 28540 for 60 sec... total: 4000 open/close in 8.84 seconds: 452.64 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.201.154@tcp:/lustre 7.4G 40M 6.9G 1% /mnt/lustre 192.168.201.154@tcp:/lustre 7.4G 40M 6.9G 1% /mnt/lustre2 PASS 35 (64s) == sanityn test 36: handle ESTALE/open-unlink correctly == 00:07:15 (1721880435) striped dir -i0 -c2 -H crush2 /mnt/lustre/d36.sanityn Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg154-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.344074 s, 152 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7561 Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 *** cycle(0) *** before(7173704) after_dd(7122504) after(7173704) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.328957 s, 159 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7561 Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 *** cycle(1) *** before(7173704) after_dd(7122504) after(7173704) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.273918 s, 191 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7561 Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 *** cycle(2) *** before(7173704) after_dd(7122504) after(7173704) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.34644 s, 151 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7561 Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 *** cycle(3) *** before(7173704) after_dd(7122504) after(7173704) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.331689 s, 158 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7561 Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 *** cycle(4) *** before(7173704) after_dd(7122504) after(7173704) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.31299 s, 168 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7561 Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 *** cycle(5) *** before(7173704) after_dd(7122504) after(7173704) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.293482 s, 179 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7561 Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 *** cycle(6) *** before(7173704) after_dd(7122504) after(7173704) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.293964 s, 178 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7561 Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 *** cycle(7) *** before(7173704) after_dd(7122504) after(7173704) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.326151 s, 161 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7561 Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 *** cycle(8) *** before(7173704) after_dd(7122504) after(7173704) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.32385 s, 162 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7561 Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 *** cycle(9) *** before(7173704) after_dd(7122504) after(7173704) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.327737 s, 160 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7561 Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 *** cycle(10) *** before(7173704) after_dd(7122504) after(7173704) PASS 36 (170s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 00:10:06 (1721880606) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d37.sanityn multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7561 - create 8296 (time 1721880617.19 total 10.00 last 829.56) total: 10000 create in 11.67 seconds: 857.02 ops/second PASS 37 (15s) == sanityn test 39a: file mtime does not change after rename ========================================================== 00:10:20 (1721880620) repeat after cancel_lru_locks PASS 39a (2s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 00:10:22 (1721880622) repeat after cancel_lru_locks PASS 39b (2s) == sanityn test 39c: check truncate mtime update ================================================================================ 00:10:25 (1721880625) repeat after cancel_lru_locks PASS 39c (2s) == sanityn test 39d: sync write should update mtime ====== 00:10:27 (1721880627) fail_loc=0x411 fail_loc=0 PASS 39d (1s) == sanityn test 40a: pdirops: create vs others ======================================================================== 00:10:28 (1721880628) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=7 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40a (8s) == sanityn test 40b: pdirops: open|create and others ======================================================================== 00:10:36 (1721880636) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=15 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (7s) == sanityn test 40c: pdirops: link and others ======================================================================== 00:10:43 (1721880643) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (7s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 00:10:50 (1721880650) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (7s) == sanityn test 40e: pdirops: rename and others ======================================================================== 00:10:57 (1721880657) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (6s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 00:11:04 (1721880664) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=8 mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (5s) == sanityn test 41b: pdirops: create vs create ======================================================================== 00:11:08 (1721880668) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 open: File exists Conflict PASS 41b (4s) == sanityn test 41c: pdirops: create vs link ======================================================================== 00:11:12 (1721880672) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (4s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 00:11:16 (1721880676) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 Conflict PASS 41d (5s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 00:11:21 (1721880681) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 41e (5s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 00:11:26 (1721880686) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 Conflict PASS 41f (5s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 00:11:31 (1721880691) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 Conflict PASS 41g (5s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 00:11:36 (1721880696) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41h (4s) == sanityn test 41i: reint_open: create vs create ======== 00:11:40 (1721880700) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (649s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 00:22:29 (1721881349) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (4s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 00:22:33 (1721881353) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 42b (4s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 00:22:37 (1721881357) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (4s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 00:22:41 (1721881361) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42d (5s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 00:22:46 (1721881366) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (5s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 00:22:51 (1721881371) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 42f (5s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 00:22:56 (1721881376) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 Conflict PASS 42g (5s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 00:23:01 (1721881381) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 42h (4s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 00:23:05 (1721881385) PASS 43a (17s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 00:23:22 (1721881402) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=48 Conflict PASS 43b (4s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 00:23:26 (1721881406) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43c (4s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 00:23:30 (1721881410) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=10 rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (4s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 00:23:34 (1721881414) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 43e (4s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 00:23:38 (1721881418) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (5s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 00:23:43 (1721881423) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 stat: cannot stat '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (4s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 00:23:47 (1721881427) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 Conflict PASS 43h (4s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 00:23:51 (1721881431) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 43i (5s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 00:23:56 (1721881436) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (35s) == sanityn test 43k: unlink vs create ==================== 00:24:31 (1721881471) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (387s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 00:30:58 (1721881858) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=2 mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Thu Jul 25 00:31:02 EDT 2024 PASS 44a (5s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 00:31:03 (1721881863) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 open: File exists Conflict PASS 44b (4s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 00:31:07 (1721881867) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (4s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 00:31:11 (1721881871) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=10 Conflict PASS 44d (5s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 00:31:16 (1721881876) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 44e (4s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 00:31:20 (1721881880) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44f (4s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 00:31:24 (1721881884) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=12 Conflict PASS 44g (4s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 00:31:28 (1721881888) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 Conflict PASS 44h (4s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 00:31:32 (1721881892) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 lfs mkdir: dirstripe error on '/mnt/lustre2/f44i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f44i.sanityn': File exists Conflict PASS 44i (5s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 00:31:37 (1721881897) PASS 45a (29s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 00:32:06 (1721881926) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=202 Conflict PASS 45b (4s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 00:32:10 (1721881930) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=7 Conflict PASS 45c (4s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 00:32:14 (1721881934) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (4s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 00:32:18 (1721881938) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=7 Conflict PASS 45e (5s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 00:32:23 (1721881943) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=12 mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (4s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 00:32:27 (1721881947) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 stat: cannot stat '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (4s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 00:32:31 (1721881951) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45h (4s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 00:32:35 (1721881955) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=7 Conflict PASS 45i (4s) == sanityn test 45j: read vs rename ====================== 00:32:39 (1721881959) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (380s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 00:38:59 (1721882339) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (5s) == sanityn test 46b: pdirops: link vs create ======================================================================== 00:39:04 (1721882344) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 open: File exists Conflict PASS 46b (5s) == sanityn test 46c: pdirops: link vs link ======================================================================== 00:39:09 (1721882349) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (4s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 00:39:13 (1721882353) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46d (5s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 00:39:18 (1721882358) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=7 Conflict PASS 46e (4s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 00:39:22 (1721882362) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=8 Conflict PASS 46f (5s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 00:39:27 (1721882367) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46g (4s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 00:39:31 (1721882371) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46h (4s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 00:39:35 (1721882375) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=7 lfs mkdir: dirstripe error on '/mnt/lustre2/f46i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f46i.sanityn': File exists Conflict PASS 46i (4s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 00:39:39 (1721882379) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 mkdir: cannot create directory '/mnt/lustre2/f47a.sanityn': File exists Conflict PASS 47a (5s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 00:39:44 (1721882384) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 open: File exists Conflict PASS 47b (5s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 00:39:49 (1721882389) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f47c.sanityn' to '/mnt/lustre2/f47c.sanityn-2': File exists Conflict PASS 47c (4s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 00:39:53 (1721882393) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 47d (4s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 00:39:57 (1721882397) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f47e.sanityn' with non-directory Conflict PASS 47e (4s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 00:40:01 (1721882401) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 Conflict PASS 47f (5s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 00:40:06 (1721882406) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 47g (4s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 00:40:10 (1721882410) 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.00279015 s, 3.7 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (7s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 00:40:17 (1721882417) 0+1 records in 0+1 records out 159 bytes (159 B) copied, 0.00424746 s, 37.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3514: kill: (23729) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 00:40:21 (1721882421) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00438523 s, 234 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0123144 s, 85.2 MB/s 3145728 PASS 51b (13s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 00:40:34 (1721882434) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00447142 s, 229 kB/s PASS 51c (6s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 00:40:41 (1721882441) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00894078 s, 117 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (3s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 00:40:43 (1721882443) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1317 0x525 0x280000401 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1317 0x525 0x280000401 read lease(1) has applied. PASS 51e (4s) == sanityn test 54: rename locking ======================= 00:40:47 (1721882447) llite.lustre-ffff8800abf93800.xattr_cache=1 llite.lustre-ffff8800b5882800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: '/mnt/lustre/d1/d2' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762669 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-25 00:40:48.000000000 -0400 Modify: 2024-07-25 00:40:48.000000000 -0400 Change: 2024-07-25 00:40:48.000000000 -0400 Birth: - ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: '/mnt/lustre/d1/d2' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281281 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-25 00:40:53.000000000 -0400 Modify: 2024-07-25 00:40:53.000000000 -0400 Change: 2024-07-25 00:40:53.000000000 -0400 Birth: - ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: '/mnt/lustre/d1/d2' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762672 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-25 00:40:58.000000000 -0400 Modify: 2024-07-25 00:40:58.000000000 -0400 Change: 2024-07-25 00:40:58.000000000 -0400 Birth: - open(O_RDWR|O_CREAT): Is a directory File: '/mnt/lustre/d1' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281282 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-25 00:40:58.000000000 -0400 Modify: 2024-07-25 00:40:59.000000000 -0400 Change: 2024-07-25 00:40:59.000000000 -0400 Birth: - ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: '/mnt/lustre/d1' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762673 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-25 00:41:04.000000000 -0400 Modify: 2024-07-25 00:41:04.000000000 -0400 Change: 2024-07-25 00:41:04.000000000 -0400 Birth: - File: '/mnt/lustre/d1/d2' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281284 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-25 00:41:04.000000000 -0400 Modify: 2024-07-25 00:41:04.000000000 -0400 Change: 2024-07-25 00:41:04.000000000 -0400 Birth: - PASS 54 (23s) == sanityn test 55a: rename vs unlink target dir ========= 00:41:10 (1721882470) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (7s) == sanityn test 55b: rename vs unlink source dir ========= 00:41:17 (1721882477) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55b.sanityn/d1/d2' to '/mnt/lustre/d55b.sanityn/d3/d2': No such file or directory PASS 55b (7s) == sanityn test 55c: rename vs unlink orphan target dir == 00:41:24 (1721882484) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (12s) == sanityn test 55d: rename file vs link ================= 00:41:36 (1721882496) fail_loc=0x155 mv: cannot move '/mnt/lustre/d55d.sanityn/f1' to '/mnt/lustre/d55d.sanityn/d55d.sanityn': Is a directory PASS 55d (9s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 00:41:45 (1721882505) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1721882505 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 20.3464 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (4s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 00:41:49 (1721882509) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: read /mnt/lustre/f56b.sanityn@33554431995904+4096 no data: Success llverdev: read /mnt/lustre/f56b.sanityn@33554431995904+4096 no data: Success llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1721882510 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 5.81306 MB/s read offset: 32768000000kB -nan MB/s llverdev: data verified successfully PASS 56b (25s) == sanityn test 60: Verify data_version behaviour ======== 00:42:14 (1721882534) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60.sanityn PASS 60 (2s) == sanityn test 70a: cd directory && rm directory ======== 00:42:16 (1721882536) PASS 70a (2s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 00:42:18 (1721882538) PASS 70b (2s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 00:42:20 (1721882540) 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00247503 s, 26.5 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00256735 s, 25.5 MB/s File: '/mnt/lustre2/f71a.sanityn' Size: 262144 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205289281307 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-25 00:42:20.000000000 -0400 Modify: 2024-07-25 00:42:20.000000000 -0400 Change: 2024-07-25 00:42:20.000000000 -0400 Birth: - 45822 extent 0 in offset 65536, length 65536 flags (0x80000000): NET extent 1 in offset 196608, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 2, sum of lengths 131072, expected sum 131072 45823 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00238058 s, 27.5 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00238165 s, 27.5 MB/s File: '/mnt/lustre/f71a.sanityn' Size: 262144 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205289281307 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-25 00:42:20.000000000 -0400 Modify: 2024-07-25 00:42:20.000000000 -0400 Change: 2024-07-25 00:42:20.000000000 -0400 Birth: - 45824 extent 0 in offset 65536, length 65536 flags (0x80000000): NET extent 1 in offset 196608, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 2, sum of lengths 131072, expected sum 131072 45823 PASS 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 00:42:22 (1721882542) 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.0034019 s, 19.3 MB/s extent 0 in offset 0, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 1, sum of lengths 65536, expected sum 65536 PASS 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 00:42:23 (1721882543) filefrag 1.46.2.wc5 (26-Mar-2022) check last flag for file with 291 extents 290: 296960.. 296963: 591872.. 591875: 4: 0000: last,net,eof check last flag for file with 292 extents 291: 297984.. 297987: 347136.. 347139: 4: 0001: last,net,eof check last flag for file with 293 extents 292: 299008.. 299011: 946176.. 946179: 4: 0000: last,net,eof PASS 71c (8s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 00:42:31 (1721882551) 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00302327 s, 21.7 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00131536 s, 49.8 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.0011966 s, 54.8 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00981922 s, 6.7 MB/s disk usage: 262144 /mnt/lustre/f71d.sanityn file size: 458752 /mnt/lustre/f71d.sanityn extent 0 in offset 0, length 65536 flags (0x80000000): NET extent 1 in offset 131072, length 65536 flags (0x80000000): NET extent 2 in offset 262144, length 65536 flags (0x80000000): NET extent 3 in offset 393216, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 4, sum of lengths 262144, expected sum 262144 PASS 71d (6s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 00:42:37 (1721882557) llite.lustre-ffff8800abf93800.xattr_cache=1 llite.lustre-ffff8800b5882800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 00:42:39 (1721882559) llite.lustre-ffff8800abf93800.xattr_cache=1 llite.lustre-ffff8800b5882800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (2s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 00:42:42 (1721882562) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00399226 s, 256 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00305828 s, 335 kB/s 6166: get lock2 6167: get lock1 6166: done 6166: get lock1 6167: done 6167: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 6166: cannot get lock1: Resource deadlock avoided fcntl cmd 7 failed: Resource deadlock avoided 6167: cannot get lock2: Resource deadlock avoided 6167: exit rc=0 6166: done 6166: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 00:42:48 (1721882568) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0135364 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00829828 s, 126 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4103: 6812 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (6s) == sanityn test 76: Verify MDT open_files listing ======== 00:42:54 (1721882574) striped dir -i0 -c2 -H crush /mnt/lustre/d76.sanityn opening files: ................ closing files: ................ PASS 76 (39s) == sanityn test 77a: check FIFO NRS policy =============== 00:43:33 (1721882613) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.159963 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0215736 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0152381 s, 68.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0229844 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132218 s, 79.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0169044 s, 62.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0214474 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0238456 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0418618 s, 25.0 MB/s , 0.0214329 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0420719 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0413337 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0455712 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0570445 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0441165 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0376472 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0386807 s, 27.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00140667 s, 745 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00858574 s, 122 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00518701 s, 202 MB/s 1048576 bytes (1.0 MB) copied, 0.00608649 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00117951 s, 889 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0204669 s, 51.2 MB/s 1+0 records in 1+0 records out , 0.00151096 s, 694 MB/s 1048576 bytes (1.0 MB) copied, 0.0186471 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0196838 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0170618 s, 61.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0143293 s, 73.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00444313 s, 236 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00662798 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00772358 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00674524 s, 155 MB/s , 0.0023107 s, 454 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=2 removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77a.sanityn' PASS 77a (3s) == sanityn test 77b: check CRR-N NRS policy ============== 00:43:36 (1721882616) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.152954 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00935169 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0146791 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122234 s, 85.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0131625 s, 79.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0406205 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0328705 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.029271 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0353943 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0466078 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.047838 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0458106 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.042108 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0509768 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0558363 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0466626 s, 22.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0586318 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00616073 s, 170 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118861 s, 88.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0130879 s, 80.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0131665 s, 79.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00790089 s, 133 MB/s , 0.00626943 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00874872 s, 120 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0135162 s, 77.6 MB/s , 0.0134443 s, 78.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00996194 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00893235 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108854 s, 96.3 MB/s , 0.0107683 s, 97.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116384 s, 90.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114666 s, 91.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120961 s, 86.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=3 removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.155293 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0157649 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0173591 s, 60.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0437748 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0507931 s, 20.6 MB/s 1048576 bytes (1.0 MB) copied, 0.0474229 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0484531 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0452104 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0511317 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0518497 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0472908 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0471319 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0554424 s, 18.9 MB/s 1048576 bytes (1.0 MB) copied, 0.0514189 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0492856 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0565033 s, 18.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0624031 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00647222 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0217945 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0190543 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00211915 s, 495 MB/s 1048576 bytes (1.0 MB) copied, 0.00148449 s, 706 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0137324 s, 76.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0249874 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0212392 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0192004 s, 54.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0234412 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0186982 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00989107 s, 106 MB/s , 0.0114618 s, 91.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00932691 s, 112 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00827182 s, 127 MB/s , 0.00941511 s, 111 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=2 removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (4s) == sanityn test 77c: check ORR NRS policy ================ 00:43:40 (1721882620) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.11013 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00857693 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.017983 s, 58.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0163497 s, 64.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109832 s, 95.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0193068 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00898378 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0327219 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0205602 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0230862 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.034081 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0359876 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0252219 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0308326 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0273439 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0195822 s, 53.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00978452 s, 107 MB/s , 0.00470254 s, 223 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00268354 s, 391 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00631391 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00879368 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00545598 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00900788 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00802625 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00707819 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100988 s, 104 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111625 s, 93.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113505 s, 92.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117874 s, 89.0 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0121042 s, 86.6 MB/s 1048576 bytes (1.0 MB) copied, 0.0121472 s, 86.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0139337 s, 75.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0573352 s, 18.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=1 removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.123568 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00939835 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110681 s, 94.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0257748 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0270746 s, 38.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0316865 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0311958 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0374005 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0372216 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0332573 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0352338 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0436631 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0412897 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0349347 s, 30.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0422568 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0407294 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0391006 s, 26.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00195291 s, 537 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0151884 s, 69.0 MB/s , 0.0224934 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00153535 s, 683 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00145465 s, 721 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00380759 s, 275 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00118442 s, 885 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0183634 s, 57.1 MB/s , 0.0211089 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00122268 s, 858 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0168252 s, 62.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0143034 s, 73.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0155409 s, 67.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120928 s, 86.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00716286 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00125383 s, 836 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=1 removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.116529 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116331 s, 90.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133614 s, 78.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0136425 s, 76.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0247311 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0299919 s, 35.0 MB/s , 0.00881635 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0302182 s, 34.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0326465 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0318522 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0146445 s, 71.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0266721 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0348583 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00720519 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0435784 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.026673 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0410304 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00909745 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126261 s, 83.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00146592 s, 715 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00161983 s, 647 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00140692 s, 745 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0123446 s, 84.9 MB/s , 0.0141462 s, 74.1 MB/s , 0.0015092 s, 695 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0172558 s, 60.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0163213 s, 64.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.022415 s, 46.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0138659 s, 75.6 MB/s , 0.0108248 s, 96.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0143315 s, 73.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0092579 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0085582 s, 123 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=2 removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (4s) == sanityn test 77d: check TRR nrs policy ================ 00:43:44 (1721882624) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.107258 s, 156 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106188 s, 98.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00924352 s, 113 MB/s 1048576 bytes (1.0 MB) copied, 0.0142151 s, 73.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133472 s, 78.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00904045 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108042 s, 97.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0311817 s, 33.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0353121 s, 29.7 MB/s 1048576 bytes (1.0 MB) copied, 0.0245065 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0384195 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0459922 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0486777 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0472833 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0534319 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0480043 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0373848 s, 28.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00194481 s, 539 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00199358 s, 526 MB/s , 0.00189989 s, 552 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00823026 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00135497 s, 774 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0099716 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00254523 s, 412 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00267752 s, 392 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106483 s, 98.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00959003 s, 109 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00306331 s, 342 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113592 s, 92.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00997836 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0164112 s, 63.9 MB/s 1048576 bytes (1.0 MB) copied, 0.0145382 s, 72.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111828 s, 93.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=2 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.095699 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00730702 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108886 s, 96.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0136354 s, 76.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00994947 s, 105 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.036222 s, 28.9 MB/s 1048576 bytes (1.0 MB) copied, 0.0427258 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0305888 s, 34.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0456581 s, 23.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0367601 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.048973 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0375706 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0392066 s, 26.7 MB/s , 0.0353998 s, 29.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0379647 s, 27.6 MB/s , 0.0350031 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0400928 s, 26.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106575 s, 98.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0124353 s, 84.3 MB/s 1048576 bytes (1.0 MB) copied, 0.01033 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107174 s, 97.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0140063 s, 74.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0119691 s, 87.6 MB/s 1048576 bytes (1.0 MB) copied, 0.0119323 s, 87.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117745 s, 89.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117728 s, 89.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104015 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01369 s, 76.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0138546 s, 75.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0123186 s, 85.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132286 s, 79.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0128498 s, 81.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122014 s, 85.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=2 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.103591 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00908966 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00955921 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0080962 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010668 s, 98.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126705 s, 82.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0187227 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0186962 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0324367 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0253923 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0353761 s, 29.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0485936 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0513619 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0409361 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0555606 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0457826 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.059962 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00921091 s, 114 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133797 s, 78.4 MB/s 1048576 bytes (1.0 MB) copied, 0.00951984 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0165792 s, 63.2 MB/s 1+0 records in 1+0 records out , 0.0103463 s, 101 MB/s 1048576 bytes (1.0 MB) copied, 0.0151456 s, 69.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107185 s, 97.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0164875 s, 63.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.014829 s, 70.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120529 s, 87.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0164427 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0207644 s, 50.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0131998 s, 79.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120271 s, 87.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0136204 s, 77.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0135036 s, 77.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=4 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (4s) == sanityn test 77e: check TBF NID nrs policy ============ 00:43:48 (1721882628) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.201.54@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0996986 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0196462 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0202582 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.015986 s, 65.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.022616 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0372274 s, 28.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0358778 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0295011 s, 35.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0278589 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0241398 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0235482 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0443686 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0296543 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0534797 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0155365 s, 67.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0525866 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0351714 s, 29.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0240215 s, 43.7 MB/s 1048576 bytes (1.0 MB) copied, 0.0184914 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0192184 s, 54.6 MB/s 1048576 bytes (1.0 MB) copied, 0.0232828 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0196138 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0236166 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0206276 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.020914 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0210078 s, 49.9 MB/s , 0.0184436 s, 56.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0268441 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0210761 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0212675 s, 49.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0208102 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0226912 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0237542 s, 44.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.101225 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00945678 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0146249 s, 71.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0211295 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0260443 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.018942 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0321913 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.023586 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0321626 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0359653 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0398191 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0446026 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0330276 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0369018 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.036816 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0255913 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.044866 s, 23.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00722091 s, 145 MB/s 1048576 bytes (1.0 MB) copied, 0.0136801 s, 76.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00945977 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00146269 s, 717 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120086 s, 87.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0216167 s, 48.5 MB/s , 0.0201023 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0015486 s, 677 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0186771 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107559 s, 97.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0206659 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00880039 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00270052 s, 388 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00873843 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00801357 s, 131 MB/s 1048576 bytes (1.0 MB) copied, 0.00588952 s, 178 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=2 removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.119577 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00818494 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0154849 s, 67.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0143759 s, 72.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0417833 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0439951 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0416621 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.038322 s, 27.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0454583 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0493487 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0413314 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0448133 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0472203 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0500063 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0435519 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0544909 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0554476 s, 18.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00974338 s, 108 MB/s 1048576 bytes (1.0 MB) copied, 0.0140411 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0156459 s, 67.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00776273 s, 135 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0138214 s, 75.9 MB/s , 0.00828033 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00857158 s, 122 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.01045 s, 100 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0104605 s, 100 MB/s , 0.0104876 s, 100 MB/s 1048576 bytes (1.0 MB) copied, 0.0104909 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116419 s, 90.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120718 s, 86.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121496 s, 86.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122262 s, 85.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0134147 s, 78.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=8 removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.103426 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111241 s, 94.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0123461 s, 84.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0221931 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0208637 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0260877 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0342261 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0402459 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0381158 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0294091 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.034335 s, 30.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0326022 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0381019 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0436052 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0460265 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0418765 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0378255 s, 27.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010779 s, 97.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00196779 s, 533 MB/s , 0.00862568 s, 122 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0217117 s, 48.3 MB/s , 0.0186448 s, 56.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0278214 s, 37.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0137208 s, 76.4 MB/s 1048576 bytes (1.0 MB) copied, 0.014784 s, 70.9 MB/s , 0.0129083 s, 81.2 MB/s , 0.0112997 s, 92.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114419 s, 91.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00889706 s, 118 MB/s 1048576 bytes (1.0 MB) copied, 0.0107878 s, 97.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00579535 s, 181 MB/s , 0.00828182 s, 127 MB/s , 0.00319434 s, 328 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' PASS 77e (5s) == sanityn test 77f: check TBF JobID nrs policy ========== 00:43:53 (1721882633) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0987501 s, 170 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126259 s, 83.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120554 s, 87.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0173078 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0206966 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0350625 s, 29.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0373788 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0458384 s, 22.9 MB/s 1048576 bytes (1.0 MB) copied, 0.037905 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0526999 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.049428 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.046611 s, 22.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0548315 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0533053 s, 19.7 MB/s , 0.0518827 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0499995 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0500693 s, 20.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=3 running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0143289 s, 73.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106551 s, 98.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107403 s, 97.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107402 s, 97.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0106248 s, 98.7 MB/s 1048576 bytes (1.0 MB) copied, 0.0105795 s, 99.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106619 s, 98.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112439 s, 93.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124809 s, 84.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0127032 s, 82.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0127724 s, 82.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0145227 s, 72.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0148317 s, 70.7 MB/s 1048576 bytes (1.0 MB) copied, 0.0153547 s, 68.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0149583 s, 70.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0179887 s, 58.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=1 removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0995278 s, 169 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126413 s, 82.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0151452 s, 69.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0343716 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0266173 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.026052 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0366262 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0384068 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0394989 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0420335 s, 24.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0450144 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0523194 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0518613 s, 20.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0639853 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0640724 s, 16.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0615922 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0672437 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00192667 s, 544 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00837505 s, 125 MB/s 1+0 records in 1+0 records out , 0.010331 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.009927 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00739575 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00968154 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00819327 s, 128 MB/s 1048576 bytes (1.0 MB) copied, 0.0181384 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00900487 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00255949 s, 410 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00542932 s, 193 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00559403 s, 187 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0068292 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.017777 s, 59.0 MB/s 1048576 bytes (1.0 MB) copied, 0.00503531 s, 208 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0158366 s, 66.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=7 removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.121794 s, 138 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00767104 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0144433 s, 72.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00884379 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00937356 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00671177 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0231015 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0140105 s, 74.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0316468 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.02386 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0293668 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0340593 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0368127 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0223771 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0255923 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0359561 s, 29.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0391913 s, 26.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00706634 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00991565 s, 106 MB/s 1048576 bytes (1.0 MB) copied, 0.0060795 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00630153 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112475 s, 93.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103341 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00842891 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00926977 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00969831 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00985154 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104568 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115649 s, 90.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109558 s, 95.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121565 s, 86.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0119427 s, 87.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122317 s, 85.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=2 removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.107738 s, 156 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0145021 s, 72.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0151103 s, 69.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.037397 s, 28.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0349205 s, 30.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0445462 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0489765 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0499945 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0453272 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0460285 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0404973 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0484555 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0437022 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0527849 s, 19.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0490889 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0509456 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0488741 s, 21.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00993468 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00745085 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00178332 s, 588 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108894 s, 96.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00663768 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0138051 s, 76.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00618533 s, 170 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00255523 s, 410 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0102614 s, 102 MB/s 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0111883 s, 93.7 MB/s , 0.00904866 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00851484 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00580934 s, 180 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00185626 s, 565 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00407806 s, 257 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00386577 s, 271 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' PASS 77f (6s) == sanityn test 77g: Change TBF type directly ============ 00:43:59 (1721882639) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (3s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 00:44:02 (1721882642) oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg154-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg154-client: oleg154-server: ssh exited with exit code 2 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Unknown error 524 oleg154-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Unknown error 524 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 12 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Unknown error 524 oleg154-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Unknown error 524 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 12 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Unknown error 524 oleg154-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Unknown error 524 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 12 PASS 77h (3s) == sanityn test 77i: Change rank of TBF rule ============= 00:44:05 (1721882645) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (6s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 00:44:11 (1721882651) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.101009 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108341 s, 96.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122747 s, 85.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0162645 s, 64.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.022657 s, 46.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.024333 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0311842 s, 33.6 MB/s 1048576 bytes (1.0 MB) copied, 0.040937 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0339233 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0394517 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0413936 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.030201 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0411635 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.035671 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0442412 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0359132 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0385153 s, 27.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00960839 s, 109 MB/s , 0.0148483 s, 70.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0132373 s, 79.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0074457 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126934 s, 82.6 MB/s , 0.0125999 s, 83.2 MB/s 1048576 bytes (1.0 MB) copied, 0.013295 s, 78.9 MB/s 1048576 bytes (1.0 MB) copied, 0.0065609 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01622 s, 64.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0164625 s, 63.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0156365 s, 67.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0167962 s, 62.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0168928 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00221588 s, 473 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00207125 s, 506 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00469578 s, 223 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.90315 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.4401 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (42s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 00:44:53 (1721882693) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg154-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg154-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg154-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg154-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.15396 s, 109 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0235096 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0246624 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0212937 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0217149 s, 48.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0263557 s, 39.8 MB/s 1048576 bytes (1.0 MB) copied, 0.0326801 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00998584 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118673 s, 88.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00992522 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00982298 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00823863 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0096797 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125171 s, 83.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0166524 s, 63.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0194937 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0610088 s, 17.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0130816 s, 80.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0131767 s, 79.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0130863 s, 80.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0139206 s, 75.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0141551 s, 74.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.014074 s, 74.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0164422 s, 63.8 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0150096 s, 69.9 MB/s , 0.0161078 s, 65.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0155531 s, 67.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.016686 s, 62.8 MB/s 1048576 bytes (1.0 MB) copied, 0.0169843 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0170359 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0181103 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0179813 s, 58.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0229973 s, 45.6 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.8337 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9482 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg154-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg154-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg154-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg154-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0891891 s, 188 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00917908 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132147 s, 79.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0110522 s, 94.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0161866 s, 64.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0152176 s, 68.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0164818 s, 63.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0200719 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00632671 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00614894 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0271295 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105384 s, 99.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00931728 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00608436 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00956171 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0394719 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0463887 s, 22.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108101 s, 97.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111341 s, 94.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010899 s, 96.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00958971 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011454 s, 91.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0123477 s, 84.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120877 s, 86.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104733 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108125 s, 97.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0103362 s, 101 MB/s 1048576 bytes (1.0 MB) copied, 0.0106588 s, 98.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0127348 s, 82.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105673 s, 99.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00899562 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117038 s, 89.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0145486 s, 72.1 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.8322 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9582 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (111s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 00:46:44 (1721882804) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg154-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg154-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg154-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg154-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0966647 s, 174 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0138843 s, 75.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132557 s, 79.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0149327 s, 70.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0219068 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0269665 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00945404 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0284614 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0411752 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00707468 s, 148 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115642 s, 90.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.019048 s, 55.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0115727 s, 90.6 MB/s 1048576 bytes (1.0 MB) copied, 0.0130516 s, 80.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0203787 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0147274 s, 71.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0084736 s, 124 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012532 s, 83.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132079 s, 79.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116165 s, 90.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0138489 s, 75.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0139517 s, 75.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0142326 s, 73.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0131769 s, 79.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.015803 s, 66.4 MB/s , 0.0130471 s, 80.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0143473 s, 73.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0137241 s, 76.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0135506 s, 77.4 MB/s , 0.0133385 s, 78.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0144337 s, 72.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01494 s, 70.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0170924 s, 61.3 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.8259 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9499 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg154-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg154-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg154-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg154-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.106234 s, 158 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0178451 s, 58.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0188673 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0214819 s, 48.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0231211 s, 45.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0221326 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00906704 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104939 s, 99.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012453 s, 84.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0351827 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0181215 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.038636 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0386317 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113787 s, 92.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00631634 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00890067 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0411391 s, 25.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113096 s, 92.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113762 s, 92.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110299 s, 95.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125213 s, 83.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122724 s, 85.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126866 s, 82.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112974 s, 92.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0127094 s, 82.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122892 s, 85.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0135798 s, 77.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125534 s, 83.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0136479 s, 76.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0134093 s, 78.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0128875 s, 81.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0128203 s, 81.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0159494 s, 65.7 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.8204 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9815 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (110s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 00:48:34 (1721882914) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.107841 s, 156 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00845505 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103144 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0146041 s, 71.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0344278 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0441712 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0429508 s, 24.4 MB/s , 0.0348658 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0373673 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0441567 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0401839 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0418053 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.051279 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0467374 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.04903 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0460782 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0488658 s, 21.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116702 s, 89.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0120158 s, 87.3 MB/s 1048576 bytes (1.0 MB) copied, 0.00322617 s, 325 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0144551 s, 72.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0142451 s, 73.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0137741 s, 76.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0196203 s, 53.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0145768 s, 71.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0153752 s, 68.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0176912 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0153676 s, 68.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0180323 s, 58.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0162062 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0165212 s, 63.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0186437 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.019527 s, 53.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.90692 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.72699 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.54@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.54@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0991885 s, 169 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00747315 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109246 s, 96.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0287275 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0296938 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0264063 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0329736 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0319301 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0304668 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0315426 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.02768 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0286684 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0280446 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.031514 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.03475 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.033984 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0361055 s, 29.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00778023 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00777878 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00592785 s, 177 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00650362 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00829571 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108459 s, 96.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00956086 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00779134 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00962142 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0097969 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00879657 s, 119 MB/s 1048576 bytes (1.0 MB) copied, 0.00910921 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00926418 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100129 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00998923 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114485 s, 91.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.89372 s, 21.4 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.7447 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.54@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.181796 s, 92.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106101 s, 98.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0168862 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0286162 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0237254 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0328304 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0415132 s, 25.3 MB/s , 0.0290508 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0322082 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.030762 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0443115 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0390488 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0330091 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0344335 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0327783 s, 32.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.037953 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0401422 s, 26.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122299 s, 85.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01019 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103606 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.014062 s, 74.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118809 s, 88.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0130042 s, 80.6 MB/s , 0.0120463 s, 87.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115302 s, 90.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012963 s, 80.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0135132 s, 77.6 MB/s , 0.0131006 s, 80.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0129587 s, 80.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0137328 s, 76.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0140976 s, 74.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0140797 s, 74.5 MB/s , 0.0142432 s, 73.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=2 removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.91151 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.87476 s, 21.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.13582 s, 124 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01132 s, 92.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.014665 s, 71.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0218179 s, 48.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0393789 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0486743 s, 21.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0362906 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0559437 s, 18.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0364405 s, 28.8 MB/s 1048576 bytes (1.0 MB) copied, 0.0371197 s, 28.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0464239 s, 22.6 MB/s 1048576 bytes (1.0 MB) copied, 0.0470416 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.049039 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0592071 s, 17.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0567522 s, 18.5 MB/s , 0.0589502 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.061573 s, 17.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113506 s, 92.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0162458 s, 64.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0155888 s, 67.3 MB/s , 0.0127723 s, 82.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0161563 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0183033 s, 57.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0140338 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0150253 s, 69.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.013924 s, 75.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0148787 s, 70.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0156995 s, 66.8 MB/s 1048576 bytes (1.0 MB) copied, 0.0162773 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0162746 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0146462 s, 71.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0158312 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0172427 s, 60.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.75467 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.73089 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.90571 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.73871 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.146659 s, 114 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0151919 s, 69.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0196489 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0203594 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0217003 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0228845 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0152501 s, 68.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0300117 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0389931 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.040889 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0365246 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0196083 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0188892 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0316221 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0429689 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00807717 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0638435 s, 16.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133998 s, 78.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0182829 s, 57.4 MB/s , 0.0133891 s, 78.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0140732 s, 74.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0178701 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0162872 s, 64.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0187873 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0170514 s, 61.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0184526 s, 56.8 MB/s 1048576 bytes (1.0 MB) copied, 0.0184697 s, 56.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0183465 s, 57.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0198261 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0201856 s, 51.9 MB/s , 0.0214117 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0203601 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0214933 s, 48.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.4651 s, 5.4 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.4465 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.156924 s, 107 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0127511 s, 82.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0129295 s, 81.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0210989 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0260033 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0273729 s, 38.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0190082 s, 55.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0194769 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0279554 s, 37.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0346679 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0240493 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0239431 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0144528 s, 72.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0244662 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0338676 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0288951 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0308411 s, 34.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00712416 s, 147 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00852227 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00324064 s, 324 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117082 s, 89.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109415 s, 95.8 MB/s 1048576 bytes (1.0 MB) copied, 0.00964083 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00585448 s, 179 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00967649 s, 108 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00419694 s, 250 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00143092 s, 733 MB/s 1048576 bytes (1.0 MB) copied, 0.00424843 s, 247 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00125865 s, 833 MB/s 1048576 bytes (1.0 MB) copied, 0.0012796 s, 819 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0129772 s, 80.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00140025 s, 749 MB/s 1048576 bytes (1.0 MB) copied, 0.00140253 s, 748 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.91325 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.7439 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.133925 s, 125 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105742 s, 99.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00953878 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0189316 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0273227 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.029242 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0226417 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0285692 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0189314 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0289329 s, 36.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0466533 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0413638 s, 25.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0360988 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0474338 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.034358 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0386229 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.054145 s, 19.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0112725 s, 93.0 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0168283 s, 62.3 MB/s , 0.0110101 s, 95.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116917 s, 89.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0170591 s, 61.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0174731 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0140511 s, 74.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0143884 s, 72.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0158614 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0160086 s, 65.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0165034 s, 63.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0167857 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0167916 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.016314 s, 64.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0172071 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0187159 s, 56.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.76338 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.74166 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.75989 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.74792 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (266s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 00:53:00 (1721883180) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (4s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 00:53:04 (1721883184) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg154-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 3.72331 s, 282 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg154-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 40.0084 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (51s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 00:53:55 (1721883235) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0976705 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118694 s, 88.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107784 s, 97.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0174091 s, 60.2 MB/s , 0.0102975 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0372443 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0347403 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0247071 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0306454 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0402582 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0295346 s, 35.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b5882800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0482816 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0390641 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0552178 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0479904 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.048551 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0609752 s, 17.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0139674 s, 75.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00590766 s, 177 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122497 s, 85.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00185003 s, 567 MB/s 1048576 bytes (1.0 MB) copied, 0.00966844 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00844197 s, 124 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0182819 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00173609 s, 604 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0156393 s, 67.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0184487 s, 56.8 MB/s 1048576 bytes (1.0 MB) copied, 0.00220163 s, 476 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00932811 s, 112 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00641283 s, 164 MB/s 1048576 bytes (1.0 MB) copied, 0.00796372 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00460418 s, 228 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00790194 s, 133 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.89784 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.97432 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.1029 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00977524 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0127621 s, 82.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0233058 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0267007 s, 39.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0186818 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0229878 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0237014 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0347525 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.033486 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0242814 s, 43.2 MB/s , 0.0302921 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0296584 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0244205 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0232263 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0334892 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104181 s, 101 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800b5882800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.018644 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0185779 s, 56.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.018478 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0213988 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0211548 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0186184 s, 56.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0187584 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01995 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.020767 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0209033 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1+0 records in 1+0 records out , 0.0208204 s, 50.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0211623 s, 49.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0216751 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0215547 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0216099 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0244568 s, 42.9 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg154-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg154-client.virtnet' removed directory: '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.89997 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.97456 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn Waiting for MDT destroys to complete oleg154-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg154-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (45s) == sanityn test 77o: Changing rank should not panic ====== 00:54:40 (1721883280) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (5s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 00:54:45 (1721883285) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (32s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 00:55:17 (1721883317) mds.MDS.mdt.nrs_policies=tbf oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg154-client: oleg154-server: ssh exited with exit code 36 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg154-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (13s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 00:55:30 (1721883330) striped dir -i0 -c1 -H all_char /mnt/lustre/d77r.sanityn mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 5778 kill md_thread xargs PASS 77r (38s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 00:56:08 (1721883368) oleg154-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg154-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg154-client: oleg154-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 00:56:11 (1721883371) striped dir -i1 -c2 -H crush /mnt/lustre/d79.sanityn File: '/mnt/lustre/d79.sanityn' Size: 8192 Blocks: 16 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798763878 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-25 00:56:12.000000000 -0400 Modify: 2024-07-25 00:56:12.000000000 -0400 Change: 2024-07-25 00:56:12.000000000 -0400 Birth: - fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 00:56:24 (1721883384) total: 10 open/close in 0.05 seconds: 192.56 ops/second multiop /mnt/lustre2/d80a.sanityn/f80a.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7561 lfs migrate: /mnt/lustre/d80a.sanityn/f80a.sanityn migrate failed: Device or resource busy (16) Finish migration, then checking.. PASS 80a (5s) == sanityn test 80b: Accessing directory during migration ========================================================== 00:56:29 (1721883389) start migration thread 17010 accessing the migrating directory for 1 minute... concurrent access 87 failures, 309 successes /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4759: 17010 Killed ( while true; do mdt_idx=$((RANDOM % MDSCOUNT)); $LFS migrate -m $mdt_idx $migdir1 &>/dev/null; done ) (wd: ~) PASS 80b (110s) == sanityn test 81a: rename and stat under striped directory ========================================================== 00:58:19 (1721883499) stat: cannot stat 'd0/0': No such file or directory File: 'd0/0' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934281 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-25 00:58:20.000000000 -0400 Modify: 2024-07-25 00:58:20.000000000 -0400 Change: 2024-07-25 00:58:20.000000000 -0400 Birth: - PASS 81a (2s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 00:58:21 (1721883501) total: 2500 open/close in 9.83 seconds: 254.35 ops/second /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4759: 19030 Killed ( while true; do touch $DIR1/$tdir; done ) (wd: ~) PASS 81b (56s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 00:59:17 (1721883557) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 00:59:18 (1721883558) - open/close 2400 (time 1721883570.94 total 10.00 last 239.95) total: 2500 open/close in 10.44 seconds: 239.57 ops/second total: 2500 open/close in 9.99 seconds: 250.25 ops/second snapshot_time 1721883611.441779327 secs.nsecs start_time 1721883558.874743057 secs.nsecs elapsed_time 52.567036270 secs.nsecs open 2505 samples [usecs] 79 762 924666 353893198 close 2504 samples [usecs] 41 1793 274761 36058273 mknod 2501 samples [usecs] 120 652 772743 248214517 unlink 2500 samples [usecs] 182 10127 1163994 690228874 mkdir 2 samples [usecs] 593 3506 4099 12643685 rmdir 2 samples [usecs] 2802 2862 5664 16042248 rename 2500 samples [usecs] 153 9271 2014567 2973476293 getattr 3767 samples [usecs] 0 1427 104023 5575615 statfs 10 samples [usecs] 8 22 131 1849 parallel_rename_file 2500 samples [usecs] 153 9271 2014567 2973476293 crossdir_rename 2500 samples [usecs] 153 9271 2014567 2973476293 snapshot_time 1721883611.442067070 secs.nsecs start_time 1721883558.874888073 secs.nsecs elapsed_time 52.567178997 secs.nsecs open 2504 samples [usecs] 62 1355 911104 344592366 close 2504 samples [usecs] 43 232 270145 30539857 mknod 2500 samples [usecs] 121 1299 762617 242940825 unlink 2500 samples [usecs] 178 11832 1183613 829895419 mkdir 2 samples [usecs] 708 7300 8008 53791264 rmdir 2 samples [usecs] 3077 8991 12068 90306010 rename 2500 samples [usecs] 156 4668 3387573 7724858807 getattr 3757 samples [usecs] 12 647 110602 4095710 statfs 10 samples [usecs] 9 22 148 2326 parallel_rename_file 2500 samples [usecs] 156 4668 3387573 7724858807 crossdir_rename 2500 samples [usecs] 156 4668 3387573 7724858807 PASS 81d (56s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 01:00:14 (1721883614) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7561 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7561 PASS 82 (2s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 01:00:16 (1721883616) start pid 1527 to create/unlink striped directory start pid 1528 to stat striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5548: 1527 Terminated ( cd $DIR1; while true; do $LFS mkdir -i1 -c2 $tdir; rmdir $tdir; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5548: 1528 Terminated ( cd $DIR2; while true; do stat $tdir > /dev/null 2>&1; done ) PASS 83 (122s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 01:02:18 (1721883738) fail_loc=0x8000060b PASS 84 (8s) == sanityn test 85: Lustre API root cache race =========== 01:02:26 (1721883746) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1721883747 Finishing test test1 at 1721883748 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 01:02:29 (1721883749) start pid 12684 to open/create under striped directory start pid 12685 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5615: 12684 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; touch $tdir/f{0..3} > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5615: 12685 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 90 (183s) == sanityn test 91: chmod and unlink striped directory === 01:05:32 (1721883932) start pid 21752 to chmod striped directory start pid 21753 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5652: 21752 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; chmod go+w $tdir > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5652: 21753 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 91 (182s) == sanityn test 92: create remote directory under orphan directory ========================================================== 01:08:34 (1721884114) fail_loc=0x1408 mkdir: cannot create directory '/mnt/lustre2/d92.sanityn/dir': No such file or directory lfs setdirstripe: cannot resolve path '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory (2) lfs setdirstripe: '/mnt/lustre2/d92.sanityn/remote_dir' is not on a Lustre filesystem: No such file or directory (2) lfs setdirstripe: cannot create dir '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory fail_loc=0 PASS 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 01:08:37 (1721884117) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5163 0x142b 0x280000400 1 5159 0x1427 0x2c0000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6342 0x18c6 0x280000401 1 6333 0x18bd 0x2c0000401 rc1=0 and rc2=0 PASS 93 (9s) == sanityn test 94: signal vs CP callback race =========== 01:08:46 (1721884126) osc.lustre-OST0000-osc-ffff8800abf93800.idle_timeout=0 osc.lustre-OST0000-osc-ffff8800b5882800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8800abf93800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8800b5882800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.015303 s, 535 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 2.99919 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5739: 5544 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00888617 s, 461 kB/s osc.lustre-OST0000-osc-ffff8800abf93800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8800b5882800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800abf93800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5882800.idle_timeout=debug PASS 94 (10s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 01:08:56 (1721884136) llite.lustre-ffff8800abf93800.fast_read=0 llite.lustre-ffff8800b5882800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0293323 s, 71.5 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8800abf93800.fast_read=1 llite.lustre-ffff8800b5882800.fast_read=1 PASS 95a (20s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 01:09:16 (1721884156) llite.lustre-ffff8800abf93800.fast_read=0 llite.lustre-ffff8800b5882800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB) copied, 0.00269406 s, 4.6 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8800abf93800.fast_read=1 llite.lustre-ffff8800b5882800.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 01:09:24 (1721884164) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 01:09:25 (1721884165) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00166413 s, 2.5 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (2s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 01:09:27 (1721884167) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0187681 s, 112 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 01:09:29 (1721884169) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.018896 s, 111 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 01:09:31 (1721884171) 1+0 records in 1+0 records out 12000 bytes (12 kB) copied, 0.00254937 s, 4.7 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 01:09:33 (1721884173) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.000948083 s, 4.3 MB/s PASS 101a (3s) == sanityn test 101b: Discard DoM data on rename ========= 01:09:36 (1721884176) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.000876547 s, 4.7 MB/s PASS 101b (3s) == sanityn test 101c: Discard DoM data on close-unlink === 01:09:39 (1721884179) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.000970197 s, 4.2 MB/s PASS 101c (3s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 01:09:42 (1721884182) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 05 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0x1c05:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.remote fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 40 02 00 00 00 74 33 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000402:0x3374:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.remote/subdir fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 06 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0x1c06:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.stripe fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 40 02 00 00 00 75 33 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000402:0x3375:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.stripe/subdir fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 40 02 00 00 00 76 33 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000402:0x3376:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! PASS 102 (4s) == sanityn test 103: Test size correctness with lockahead ========================================================== 01:09:46 (1721884186) striped dir -i1 -c2 -H crush /mnt/lustre/d103.sanityn fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1721884187 lockahead_test: lockahead_test.c:1191: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1721884189 Finishing test test23 at 1721884190 PASS 103 (7s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 01:09:54 (1721884194) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00427016 s, 240 kB/s oleg154-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg154-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps STAT a|m|ctime 1721884195 1721884195 1721884195 MDT a|m|ctime 1721884195 1721884195 1721884195 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00450249 s, 227 kB/s oleg154-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg154-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps STAT a|m|ctime 1721884195 1721884197 1721884197 MDT a|m|ctime 1721884195 1721884197 1721884197 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0112944 s, 90.7 kB/s oleg154-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg154-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps STAT a|m|ctime 1721884195 1721884202 1721884202 MDT a|m|ctime 1721884195 1721884202 1721884202 oleg154-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg154-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps oleg154-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg154-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps STAT a|m|ctime 1721884195 1721884202 1721884202 MDT a|m|ctime 1721884195 1721884202 1721884202 oleg154-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg154-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps STAT a|m|ctime 1721884195 1721884202 1721884211 MDT a|m|ctime 1721884195 1721884202 1721884211 PASS 104 (20s) == sanityn test 105: Glimpse and lock cancel race ======== 01:10:13 (1721884213) striped dir -i1 -c2 -H all_char /mnt/lustre/d105.sanityn fail_loc=0x416 PASS 105 (28s) == sanityn test 106a: Verify the btime via statx() ======= 01:10:41 (1721884241) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 01:10:42 (1721884242) SKIP: sanityn test_106b statx() only test SKIP 106b (1s) == sanityn test 106c: Verify statx attributes mask ======= 01:10:43 (1721884243) SKIP: sanityn test_106c statx() only test SKIP 106c (1s) == sanityn test 107a: Basic grouplock conflict =========== 01:10:44 (1721884244) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0905201 s, 116 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6338 0x18c2 0x2c0000401 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7561 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7561 First grouplock blocks second one PASS 107a (5s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 01:10:49 (1721884249) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.090101 s, 116 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6348 0x18cc 0x280000401 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7561 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (8s) == sanityn test 108a: lseek: parallel updates ============ 01:10:57 (1721884257) 1+0 records in 1+0 records out 10000 bytes (10 kB) copied, 0.0029735 s, 3.4 MB/s fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB) copied, 4.20515 s, 2.0 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 01:11:04 (1721884264) 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 2 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 6 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 16 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 19 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 21 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 26 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 27 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 28 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 29 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 31 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 37 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 40 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 41 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 43 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 46 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 47 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: libkmod: kmod_module_get_refcnt: could not read integer from '/sys/module/acpi_cpufreq/refcnt': 'No such device' oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 49 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 50 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre 192.168.201.154@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre2 PASS 109 (447s) == sanityn test 110: do not grant another lock on resend ========================================================== 01:18:31 (1721884711) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: '/mnt/lustre/f110.sanityn' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238843711491 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-25 01:18:31.000000000 -0400 Modify: 2024-07-25 01:18:31.000000000 -0400 Change: 2024-07-25 01:18:31.000000000 -0400 Birth: - fail_loc=0 fail_val=0 Connected clients: oleg154-client.virtnet oleg154-client.virtnet PASS 110 (65s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 01:19:36 (1721884776) fail_loc=0x8000018a /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6528: 8727 Terminated $MULTIOP $DIR2/$tdir/mdt0dir/foodir/file2 Ow4096_c PASS 111 (6s) == sanityn test 112: update max-inherit in default LMV === 01:19:42 (1721884782) lmv.lustre-clilmv-ffff8800a99fc800.qos_threshold_rr=90 lmv.lustre-clilmv-ffff8800abf9c800.qos_threshold_rr=90 PASS 112 (4s) == sanityn test 113: check servers of specified fs ======= 01:19:46 (1721884786) lustre-OST0000-osc-ffff8800abf9c800 active. lustre-OST0001-osc-ffff8800abf9c800 active. lustre-OST0000-osc-ffff8800abf9c800 active. lustre-OST0001-osc-ffff8800abf9c800 active. MGC192.168.201.154@tcp active. PASS 113 (3s) == sanityn test 114: implicit default LMV inherit ======== 01:19:49 (1721884789) /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 PASS 114 (8s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 01:19:57 (1721884797) total: 3000 create in 5.87 seconds: 510.77 ops/second oleg154-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg154-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps oleg154-server: htree_dump_int_block: Ext2 directory block not found while calculating checksum for htree root oleg154-server: USE: f267 f846 fail_loc=0x80002401 fail_val=5 mkdir: cannot create directory '/mnt/lustre/d115.sanityn/f846': File exists PASS 115 (15s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 01:20:12 (1721884812) llite.lustre-ffff8800a99fc800.intent_mkdir=1 llite.lustre-ffff8800abf9c800.intent_mkdir=1 MD layout /mnt/lustre/d116.sanityn: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000bd0:0x697:0x0] 1 [0x240000404:0x67f:0x0] mkdir /mnt/lustre/d116.sanityn/tdir0 setdirstripe -D -i1 /mnt/lustre2/d116.sanityn/tdir0 mkdir /mnt/lustre/d116.sanityn/tdir0/tdir11 lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none mdtidx FID[seq:oid:ver] lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none mdtidx FID[seq:oid:ver] PASS 116 (3s) cleanup: ====================================================== == sanityn test complete, duration 5076 sec ============== 01:20:16 (1721884816) === sanityn: start cleanup 01:20:16 (1721884816) === Stopping clients: oleg154-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg154-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 01:22:00 (1721884920) ===