-----============= acceptance-small: sanityn ============----- Thu May 1 23:28:00 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg344-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanityn.*ex: No such file or directory excepting tests: 27 28 skipping tests SLOW=no: 33a === sanityn: start setup 23:28:10 (1746156490) === Starting client oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Started clients oleg344-client.virtnet: 192.168.203.144@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg344-client.virtnet: executing check_config_client /mnt/lustre oleg344-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg344-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800ae0f0000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8800b60dc000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800ae0f0000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b60dc000.idle_timeout=debug disable quota as required oleg344-server: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanityn: finish setup 23:28:25 (1746156505) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0167288 s, 62.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7670] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 23:28:26 (1746156506) PASS 0 (2s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 23:28:28 (1746156508) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (2s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 23:28:30 (1746156510) -rw-r--r-- 1 root root 0 May 1 23:28 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (2s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 23:28:33 (1746156513) -rw-r--r-- 1 root root 0 May 1 23:28 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (2s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 23:28:35 (1746156515) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (2s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 23:28:37 (1746156517) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (3s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 23:28:40 (1746156520) total 0 -rwxrwxrwx 1 root root 0 May 1 23:28 f2a -rwxrwxrwx 1 root root 0 May 1 23:28 f2b total 0 -rwxrwxrwx 1 root root 0 May 1 23:28 f2a -rwxrwxrwx 1 root root 0 May 1 23:28 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.00554896 s, 92.3 kB/s PASS 2e (2s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 23:28:42 (1746156522) /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has type file OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has perms 0777 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by user #500 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by group #500 OK Can't lstat /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn: No such file or directory PASS 2f (2s) == sanityn test 2g: check blocks update on sync write ==== 23:28:44 (1746156524) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.184967 s, 11.3 MB/s /mnt/lustre/f2g.sanityn has 4096 blocks /mnt/lustre2/f2g.sanityn has 4096 blocks PASS 2g (3s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 23:28:47 (1746156527) PASS 3 (3s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 23:28:50 (1746156530) PASS 4 (3s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 23:28:53 (1746156533) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (2s) == sanityn test 6: remove of open file on other node ============================================================================ 23:28:55 (1746156535) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (3s) == sanityn test 7: remove of open directory on other node ======================================================================= 23:28:58 (1746156538) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (1s) == sanityn test 8: remove of open special file on other node ==================================================================== 23:29:00 (1746156540) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (2s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 23:29:02 (1746156542) PASS 9a (1s) == sanityn test 9b: append to striped sparse file ======== 23:29:04 (1746156544) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0134072 s, 78.2 MB/s 3+0 records in 3+0 records out 3 bytes (3 B) copied, 0.0136637 s, 0.2 kB/s Data read (expecting 'foo'): foo PASS 9b (1s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 23:29:05 (1746156545) 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00264207 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00506627 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00431145 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00423119 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00840337 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00908011 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00535688 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00699539 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00587041 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00708311 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00724552 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00704422 s, 0.1 kB/s PASS 10a (3s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 23:29:08 (1746156548) 1+0 records in 1+0 records out 3072 bytes (3.1 kB) copied, 0.0100265 s, 306 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0170953 s, 240 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB) copied, 0.000900337 s, 3.4 MB/s PASS 10b (3s) == sanityn test 11: execution of file opened for write should return error ============================================================== 23:29:11 (1746156551) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d11 multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7670 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 301: /mnt/lustre2/d11/f: Text file busy PASS 11 (2s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 23:29:13 (1746156553) warning: '-runtime' deprecated, use '-t runtime' instead - link 704 (time 1746156564.73 total 10.01 last 70.32) - link 1813 (time 1746156574.73 total 20.01 last 110.88) - link 2940 (time 1746156584.73 total 30.02 last 112.66) - link 3799 (time 1746156594.73 total 40.02 last 85.87) - link 4748 (time 1746156604.74 total 50.02 last 94.86) - link 5572 (time 1746156614.74 total 60.03 last 82.38) - link 6520 (time 1746156624.75 total 70.03 last 94.77) - link 7561 (time 1746156634.75 total 80.04 last 104.02) - link 8509 (time 1746156644.76 total 90.04 last 94.74) total: 9444 link in 100.01 seconds: 94.43 ops/second using seed 4019026614 running for 100 seconds - stat 10000 (time 1746156560 ; total 5 ; last 5) - stat 20000 (time 1746156565 ; total 10 ; last 5) - stat 30000 (time 1746156569 ; total 14 ; last 4) - stat 40000 (time 1746156572 ; total 17 ; last 3) - stat 50000 (time 1746156576 ; total 21 ; last 4) - stat 60000 (time 1746156580 ; total 25 ; last 4) - stat 70000 (time 1746156584 ; total 29 ; last 4) - stat 80000 (time 1746156588 ; total 33 ; last 4) - stat 90000 (time 1746156592 ; total 37 ; last 4) - stat 100000 (time 1746156596 ; total 41 ; last 4) - stat 110000 (time 1746156600 ; total 45 ; last 4) - stat 120000 (time 1746156604 ; total 49 ; last 4) - stat 127401 (time 1746156615 ; total 60 ; last 11) - stat 130000 (time 1746156620 ; total 65 ; last 5) - stat 135938 (time 1746156631 ; total 76 ; last 11) - stat 140000 (time 1746156637 ; total 82 ; last 6) - stat 145156 (time 1746156648 ; total 93 ; last 11) total: 148570 stats in 100 seconds: 1485.699951 stats/second - unlinked 0 (time 1746156605 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile9444) error: No such file or directory total: 9443 unlinks in 101 seconds: 93.495049 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (23402) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (23405) - No such process PASS 12 (155s) == sanityn test 13: test directory page revocation ======= 23:31:48 (1746156708) striped dir -i1 -c2 -H crush2 /mnt/lustre/d13 f13 f13 PASS 13 (4s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 23:31:52 (1746156712) striped dir -i0 -c2 -H crush2 /mnt/lustre/d14aa.sanityn multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7670 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 336: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 24832 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (3s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 23:31:55 (1746156715) striped dir -i0 -c2 -H all_char /mnt/lustre/d14ab.sanityn open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 25480 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (3s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 23:31:58 (1746156718) striped dir -i0 -c2 -H crush /mnt/lustre/d14b.sanityn truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 351: 26127 Terminated $DIR1/$tdir/sleep 60 PASS 14b (3s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 23:32:01 (1746156721) striped dir -i0 -c2 -H crush /mnt/lustre/d14c.sanityn cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 363: 26774 Terminated $DIR1/$tdir/sleep 60 PASS 14c (4s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 23:32:05 (1746156725) striped dir -i0 -c2 -H crush /mnt/lustre/d14d.sanityn chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 374: 27423 Terminated $DIR1/$tdir/sleep 60 PASS 14d (4s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 23:32:09 (1746156729) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_CODE=117440512 CLIENT_OS_VERSION_ID=7 STRIPECOUNT=2 ORIGFREE=7207396 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7207396kB free gt MAXFREE 800000kB, increase 800000 (or reduce test fs size) to proceed Waiting for MDT destroys to complete SKIP 15 (11s) == sanityn test 16a: 2500 iterations of dual-mount fsx === 23:32:20 (1746156740) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.23086 s, 36.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.194355 s, 43.2 MB/s Chance of close/open is 1 in 50 Seed set to 6742 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn truncating to largest ever: 0x5e570b truncating to largest ever: 0x63a7e6 fallocating to largest ever: 0x836f8b truncating to largest ever: 0x923866 000100[0] 1746156745.336198 fallocate 0x1bbbf2 thru 0x1c16f8 (0x5b07 bytes) truncating to largest ever: 0x97363f 000200[0] 1746156748.043393 read 0x6dccf2 thru 0x6e24aa (0x57b9 bytes) 000300[1] 1746156750.815632 write 0x744a78 thru 0x750cdc (0xc265 bytes) fallocating to largest ever: 0x99c057 fallocating to largest ever: 0x9a2525 truncating to largest ever: 0x9d5fa2 fallocating to largest ever: 0x9f9dd4 000500[0] 1746156755.953544 mapread 0x131629 thru 0x13e23d (0xcc15 bytes) 000600[1] 1746156758.351025 punch from 0x4fefc1 to 0x50946f (0xa4af bytes) 000700[0] 1746156761.226606 trunc from 0x4ddd20 to 0x4e5718 (0x79f9 bytes) 000800[0] 1746156763.411751 read 0x09b2e2 thru 0x0a272a (0x7449 bytes) 001000[1] 1746156768.184798 punch from 0x59d454 to 0x5ab146 (0xdcf3 bytes) 001100[0] 1746156770.544897 trunc from 0x42cd70 to 0x0ff2a0 (0x32dacf bytes) 001200[1] 1746156772.524987 fallocate 0x396629 thru 0x39734a (0xd22 bytes) 001300[0] 1746156775.211952 mapread 0x7090f6 thru 0x70ae0c (0x1d17 bytes) 001400[0] 1746156777.742166 fallocate 0x066ef9 thru 0x06c283 (0x538b bytes) 001600[0] 1746156782.860451 mapwrite 0x18b105 thru 0x193fa4 (0x8ea0 bytes) 001700[1] 1746156784.880735 trunc from 0x6f3ef0 to 0x6ff592 (0xb6a3 bytes) 001800[1] 1746156787.295996 read 0x21c47c thru 0x22a522 (0xe0a7 bytes) 001900[0] 1746156789.962241 punch from 0x821f50 to 0x82bdb4 (0x9e65 bytes) 002000[0] 1746156792.374843 fallocate 0x224764 thru 0x2261e3 (0x1a80 bytes) 002200[0] 1746156798.020993 mapread 0x62f055 thru 0x63a011 (0xafbd bytes) 002300[1] 1746156800.581554 mapwrite 0x1cc6e6 thru 0x1d3647 (0x6f62 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 6805 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fallocating to largest ever: 0x1451d7 fallocating to largest ever: 0x8f48b3 fallocating to largest ever: 0x94ca56 fallocating to largest ever: 0x9de1de 000100[1] 1746156807.946077 punch from 0x3efb2b to 0x3ff14c (0xf622 bytes) 000200[1] 1746156809.642260 write_OD 0x050000 thru 0x05afff (0xb000 bytes) 000300[0] 1746156811.321134 trunc from 0x77ac49 to 0x784ebb (0xa273 bytes) 000400[1] 1746156813.416567 read_OD 0x77b000 thru 0x786fff (0xc000 bytes) 000500[0] 1746156815.361808 write_OD 0x679000 thru 0x67cfff (0x4000 bytes) truncating to largest ever: 0x9ed05a 000600[0] 1746156817.321279 mapread 0x2d0000 thru 0x2d89f0 (0x89f1 bytes) 000700[1] 1746156819.410297 trunc from 0x68638d to 0x68e6a2 (0x8316 bytes) 000800[1] 1746156821.883094 trunc from 0x2d95a9 to 0x6f4827 (0x41b27f bytes) 000900[0] 1746156824.188221 punch from 0x3643fe to 0x3706f4 (0xc2f7 bytes) 001000[1] 1746156826.381605 write 0x28d000 thru 0x292579 (0x557a bytes) 001100[1] 1746156828.640555 mapread 0x375000 thru 0x37a67f (0x5680 bytes) 001200[1] 1746156830.651911 mapread 0x2e2000 thru 0x2e457e (0x257f bytes) 001300[1] 1746156832.815209 mapread 0x048000 thru 0x0540b6 (0xc0b7 bytes) 001500[1] 1746156837.393017 mapread 0x010000 thru 0x012cae (0x2caf bytes) 001600[1] 1746156840.040735 mapread 0x70b000 thru 0x71300d (0x800e bytes) 001800[1] 1746156844.474347 trunc from 0x400883 to 0x4079fa (0x7178 bytes) 002000[0] 1746156848.955177 mapwrite 0x4e1000 thru 0x4ef57e (0xe57f bytes) 002100[0] 1746156850.796131 trunc from 0x9aaff7 to 0x5c75c9 (0x3e3a2d bytes) 002300[1] 1746156855.230068 mapwrite 0x411000 thru 0x41c5f5 (0xb5f6 bytes) 002400[1] 1746156857.551978 read 0x133000 thru 0x1381df (0x51e0 bytes) 002500[1] 1746156859.678633 trunc from 0x81a000 to 0x42fd80 (0x3ea27f bytes) All operations completed A-OK! PASS 16a (121s) == sanityn test 16b: 2500 iterations of dual-mount fsx at small size ========================================================== 23:34:21 (1746156861) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.196704 s, 42.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.21201 s, 39.6 MB/s Chance of close/open is 1 in 50 Seed set to 6863 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn skipping zero size read skipping zero size read skipping zero length punch hole skipping zero size read skipping zero length zero range truncating to largest ever: 0x826ae9 fallocating to largest ever: 0x9f504f 000100[0] 1746156866.672828 trunc from 0x155b7e to 0x156dbc (0x123f bytes) 000200[1] 1746156869.172697 read 0x2a53c5 thru 0x2a6f41 (0x1b7d bytes) 000300[0] 1746156871.876288 mapwrite 0x99b53b thru 0x99c773 (0x1239 bytes) 000400[1] 1746156874.444039 read 0x60e095 thru 0x60edd3 (0xd3f bytes) 000500[0] 1746156877.329749 mapread 0x0bbcf5 thru 0x0bd95c (0x1c68 bytes) fallocating to largest ever: 0x9febd0 000700[1] 1746156882.606084 mapwrite 0x3bb13e thru 0x3bbdc2 (0xc85 bytes) 000800[1] 1746156885.354009 mapwrite 0x4384b5 thru 0x439afb (0x1647 bytes) 000900[1] 1746156888.399941 punch from 0x22e2b8 to 0x22fea0 (0x1be9 bytes) 001000[0] 1746156890.854275 trunc from 0x6db5d2 to 0x6dcc8a (0x16b9 bytes) 001100[0] 1746156893.156712 fallocate 0x1826c0 thru 0x182f26 (0x867 bytes) 001200[0] 1746156895.271450 trunc from 0x4d8093 to 0x4d8222 (0x190 bytes) 001500[0] 1746156902.505576 mapwrite 0x173798 thru 0x174460 (0xcc9 bytes) 001600[1] 1746156904.809665 read 0x0acece thru 0x0ade97 (0xfca bytes) 001700[0] 1746156907.884940 trunc from 0x40ebf6 to 0x104ad3 (0x30a122 bytes) 001900[0] 1746156913.340910 read 0x01ef84 thru 0x01f139 (0x1b6 bytes) 002200[0] 1746156921.009804 trunc from 0x60220c to 0x1c47a4 (0x43da67 bytes) 002300[1] 1746156923.782056 punch from 0x4d3949 to 0x4d5460 (0x1b18 bytes) 002400[1] 1746156926.571657 punch from 0x2c291d to 0x2c34b4 (0xb98 bytes) All operations completed A-OK! PASS 16b (70s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 23:35:32 (1746156932) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.247703 s, 33.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.198505 s, 42.3 MB/s Waiting for MDT destroys to complete Chance of close/open is 1 in 50 Seed set to 6946 fd 0: /mnt/lustre/f16c.sanityn fd 1: /mnt/lustre2/f16c.sanityn skipping zero length zero range skipping zero size read skipping zero length zero range skipping zero size read skipping zero size read skipping zero length zero range skipping zero length punch hole truncating to largest ever: 0x678293 fallocating to largest ever: 0x7accc6 zero_range to largest ever: 0x7c6789 truncating to largest ever: 0x98b112 fallocating to largest ever: 0x99233f 000100[1] 1746156949.804368 trunc from 0x59cd9f to 0x5a6c91 (0x9ef3 bytes) truncating to largest ever: 0x9f463f 000300[0] 1746156954.553747 mapwrite 0x3fdeda thru 0x40a79c (0xc8c3 bytes) 000400[0] 1746156956.965761 mapwrite 0x41a224 thru 0x427288 (0xd065 bytes) truncating to largest ever: 0x9f6347 truncating to largest ever: 0x9f9408 000700[0] 1746156966.291803 mapwrite 0x635e0b thru 0x638eee (0x30e4 bytes) 000800[1] 1746156969.397887 mapread 0x053c6e thru 0x059e8c (0x621f bytes) 001200[0] 1746156979.854095 trunc from 0x95dbf3 to 0x3f5b85 (0x56806d bytes) 001300[1] 1746156982.357523 trunc from 0x4756ea to 0x7aa2f8 (0x334c0f bytes) fallocating to largest ever: 0xa00000 001500[1] 1746156987.571250 read 0x5895fc thru 0x589bdd (0x5e2 bytes) 001600[0] 1746156989.691942 trunc from 0x5bc2d1 to 0x5c38cd (0x75fd bytes) 001700[0] 1746156992.507191 trunc from 0x960833 to 0x72b055 (0x2357dd bytes) 001800[1] 1746156995.723965 trunc from 0x8cb759 to 0x61adfc (0x2b095c bytes) 001900[0] 1746156998.437496 trunc from 0x59024e to 0x5852ff (0xaf4e bytes) 002000[0] 1746157001.499085 trunc from 0x9f05e2 to 0x9da6f9 (0x15ee8 bytes) 002100[1] 1746157004.007672 punch from 0x4e4ad8 to 0x4ebb81 (0x70aa bytes) 002200[1] 1746157006.827922 fallocate 0x60ae0d thru 0x611f7b (0x716f bytes) 002300[1] 1746157009.943939 punch from 0x3b6547 to 0x3bdea1 (0x795b bytes) 002500[0] 1746157016.267067 fallocate 0x29b2a1 thru 0x2aa071 (0xedd1 bytes) All operations completed A-OK! PASS 16c (88s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 23:36:59 (1746157019) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.239663 s, 35.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.210829 s, 39.8 MB/s ldlm.namespaces.MGC192.168.203.144@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800ae0f0000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b60dc000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800ae0f0000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b60dc000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b60dc000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b60dc000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 3.55685 s, 29.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 1.53533 s, 68.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 1.6184 s, 64.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 3.60834 s, 29.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 2.6302 s, 39.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 1.83625 s, 57.1 MB/s PASS 16d (21s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 23:37:20 (1746157040) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.104178 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0218505 s, 48.0 MB/s PASS 16e (4s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 23:37:24 (1746157044) TIMEOUT PASS 16f (24s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 23:37:48 (1746157068) TIMEOUT PASS 16g (24s) == sanityn test 16h: mmap read after truncate file ======= 23:38:12 (1746157092) striped dir -i0 -c2 -H crush2 /mnt/lustre/d16h.sanityn ------- before truncate -------- 0003220 3035 3a31 3035 3a31 2f3a 6f68 656d 732f 0003240 6e61 7469 7579 7273 3a31 622f 6e69 622f 0003260 7361 0a68 0003264 ------- after truncate -------- 0001460 7373 6468 783a 373a 3a34 3437 503a 6972 0001500 6976 656c 6567 732d 7065 7261 7461 6465 0001520 5320 4853 2f3a 6176 2f72 0001532 PASS 16h (4s) == sanityn test 16i: read after truncate file ============ 23:38:16 (1746157096) striped dir -i0 -c2 -H crush2 /mnt/lustre/d16i.sanityn ------- before truncate -------- 0003220 3035 3a31 3035 3a31 2f3a 6f68 656d 732f 0003240 6e61 7469 7579 7273 3a31 622f 6e69 622f 0003260 7361 0a68 0003264 ------- after truncate -------- 0001460 7373 6468 783a 373a 3a34 3437 503a 6972 0001500 6976 656c 6567 732d 7065 7261 7461 6465 0001520 5320 4853 2f3a 6176 2f72 0001532 PASS 16i (4s) == sanityn test 16j: race dio with buffered i/o ========== 23:38:20 (1746157100) 1+0 records in 1+0 records out 26214400 bytes (26 MB) copied, 0.619038 s, 42.3 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.55492 s, 47.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.926436 s, 28.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.431292 s, 60.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.883065 s, 29.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 dd: error reading '/mnt/lustre/f16j.sanityn.1': Invalid argument 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0719947 s, 0.0 kB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.41809 s, 62.7 MB/s sanityn test_16j: @@@@@@ FAIL: dio copy 1 w/bsize 1049600 failed: 1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:760:test_16j() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:830:main() Dumping lctl log to /tmp/testlogs//sanityn.test_16j.*.1746157108.log rsync: chown "/tmp/testlogs/.sanityn.test_16j.debug_log.oleg344-server.1746157108.log.Y6HFlW" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanityn.test_16j.dmesg.oleg344-server.1746157108.log.4hYtLD" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 16j (11s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 23:38:32 (1746157112) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 skipping zero size read truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Thu May 1 23:38:54 EDT 2025 All operations completed A-OK! All operations completed A-OK! + fsx 2 done: rc=0 Thu May 1 23:38:54 EDT 2025 + fsx 3 done: rc=0 Thu May 1 23:38:54 EDT 2025 All operations completed A-OK! + fsx 4 done: rc=0 Thu May 1 23:38:55 EDT 2025 + fsx 5 done: rc=0 Thu May 1 23:38:55 EDT 2025 PASS 16k (29s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 23:39:01 (1746157141) fail_loc=0x8000030a PASS 17 (5s) == sanityn test 18: mmap sanity check =========================================================================================== 23:39:06 (1746157146) mmap test1: basic mmap operation (PASS, 0.051054s) mmap test2: MAP_PRIVATE not write back (PASS, 0.045134s) mmap test3: concurrent mmap ops on two nodes (PASS, 18.69s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.2936s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.1334s) mmap test6: check mmap write/read content on two nodes (PASS, 0.18814s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.94262s) mmap test11: multi-thread shared mmap (PASS, 0.62864s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0743s) PASS 18 (31s) == sanityn test 19: test concurrent uncached read races ========================================================================= 23:39:37 (1746157177) 32+0 records in 32+0 records out 16777216 bytes (17 MB) copied, 0.27367 s, 61.3 MB/s loop 5 loop 10 loop 15 loop 20 PASS 19 (15s) == sanityn test 20: test extra readahead page left in cache ============================================================== 23:39:53 (1746157193) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d20.sanityn PASS 20 (4s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 23:39:56 (1746157196) striped dir -i1 -c2 -H crush2 /mnt/lustre/d21.sanityn rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (3s) == sanityn test 23: others should see updated atime while another read============================================================== 23:39:59 (1746157199) now is 1746157200 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7670 new atime is 1746157261 PASS 23 (64s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 23:41:03 (1746157263) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 19588 3587432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 44764 3562256 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 64352 7149688 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2060 1285628 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 19588 3587432 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 44764 3562256 2% /mnt/lustre2[OST:1] filesystem_summary: 7666232 64352 7149688 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 351 999.7K 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1000.0K 279 999.7K 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 256.0K 379 255.6K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 256.0K 376 255.6K 1% /mnt/lustre[OST:1] filesystem_summary: 511.9K 630 511.3K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 351 999.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1000.0K 279 999.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 256.0K 379 255.6K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 256.0K 376 255.6K 1% /mnt/lustre2[OST:1] filesystem_summary: 511.9K 630 511.3K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 1.3G 2.0M 1.2G 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1.3G 1.8M 1.2G 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3.7G 19.1M 3.4G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.7G 43.7M 3.4G 2% /mnt/lustre[OST:1] filesystem_summary: 7.3G 62.8M 6.8G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 351 1023649 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1024000 279 1023721 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 262144 379 261765 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 262144 376 261768 1% /mnt/lustre2[OST:1] filesystem_summary: 524163 630 523533 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 19588 3587432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 44764 3562256 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 64352 7149688 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 351 999.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1000.0K 279 999.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 256.0K 379 255.6K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 256.0K 376 255.6K 1% /mnt/lustre2[OST:1] filesystem_summary: 511.9K 630 511.3K 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 351 1023649 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1024000 279 1023721 1% /mnt/lustre[MDT:1] filesystem_summary: 2048000 630 2047370 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 351 1023649 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1024000 279 1023721 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 262144 379 261765 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 262144 376 261768 1% /mnt/lustre2[OST:1] filesystem_summary: 524163 630 523533 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 19588 3587432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 44764 3562256 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 64352 7149688 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2060 1285628 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 19588 3587432 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 44764 3562256 2% /mnt/lustre2[OST:1] filesystem_summary: 7666232 64352 7149688 1% /mnt/lustre2 PASS 24a (3s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 23:41:06 (1746157266) PASS 24b (3s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 23:41:09 (1746157269) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (4s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 23:41:13 (1746157273) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] PASS 25b (3s) == sanityn test 26a: allow mtime to get older ============ 23:41:16 (1746157276) utime: good mknod times 1746157275 <= 1746157276 <= 1746157276 for /mnt/lustre/f26a utime: good mknod times 1746157275 <= 1746157276 <= 1746157276 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (3s) == sanityn test 26b: sync mtime between ost and mds ====== 23:41:19 (1746157279) PASS 26b (5s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 23:41:24 (1746157284) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7670 File: '/mnt/lustre/f26c.sanityn' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279601 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-05-01 23:41:24.000000000 -0400 Birth: - File: '/mnt/lustre2/f26c.sanityn' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279601 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-05-01 23:41:24.000000000 -0400 Birth: - PASS 26c (3s) SKIP: sanityn test_27 skipping excluded test 27 SKIP: sanityn test_28 skipping ALWAYS excluded test 28 == sanityn test 30: recreate file race =================== 23:41:29 (1746157289) striped dir -i0 -c2 -H crush /mnt/lustre/d30.sanityn 0 PASS 30 (5s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 23:41:33 (1746157293) striped dir -i1 -c2 -H crush2 /mnt/lustre/d31a.sanityn fail_loc=0x314 PASS 31a (4s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 23:41:37 (1746157297) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg344-server mds-ost sync done. Waiting for MDT destroys to complete striped dir -i1 -c2 -H crush2 /mnt/lustre/d31b.sanityn fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.144@tcp:/lustre 7666232 64352 7148692 1% /mnt/lustre2 PASS 31b (10s) == sanityn test 31r: open-rename(replace) race =========== 23:41:47 (1746157307) -rw-r--r-- 1 root root 0 May 1 23:41 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (6s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 23:41:53 (1746157313) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1213: 18018 Terminated tail -f $tfile PASS 31s (4s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 23:41:57 (1746157317) PASS 31t (4s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 23:42:02 (1746157322) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (2s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 23:42:04 (1746157324) Stopping /mnt/lustre-mds1 (opts:) on oleg344-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg344-server: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-MDT0000 create file on 2nd stripe under striped directory /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre/d33c.sanityn/striped touch /mnt/lustre2/d33c.sanityn/striped/subfile SoLC count 2 Below operations shouldn't trigger Solc: create file on 2nd stripe after setattr chmod 777 /mnt/lustre/d33c.sanityn/striped; touch /mnt/lustre2/d33c.sanityn/striped/subfile create remote dir and local dir /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote create local dir after remote dir creation transaction commit /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote.3; do_facet mds2 /home/green/git/lustre-release/lustre/utils/lctl set_param -n osd*.*MDT0001.force_sync 1; mkdir /mnt/lustre/d33c.sanityn/local.3 Solc count 0 PASS 33c (22s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 23:42:26 (1746157346) remote directory create /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre2/d33d.sanityn/subdir CoS count 2 cross-MDT rename mkdir /mnt/lustre/d33d.sanityn; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn/d1; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33d.sanityn/d2; touch /mnt/lustre/d33d.sanityn/d1/src mv /mnt/lustre2/d33d.sanityn/d1/src /mnt/lustre2/d33d.sanityn/d2/tgt CoS count 2 migrate /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs migrate -m 1 /mnt/lustre2/d33d.sanityn CoS count 2 PASS 33d (21s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 23:42:47 (1746157367) plain dir creation mkdir /mnt/lustre2/d33e.sanityn/plain open file and write echo abc > /mnt/lustre2/d33e.sanityn/f33e.sanityn append write echo dfg >> /mnt/lustre2/d33e.sanityn/f33e.sanityn setattr touch /mnt/lustre2/d33e.sanityn/f33e.sanityn file unlink rm /mnt/lustre2/d33e.sanityn/f33e.sanityn plain dir unlink rmdir /mnt/lustre2/d33e.sanityn/plain striped directory creation /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre2/d33e.sanityn/striped set default LMV to create striped subdir /home/green/git/lustre-release/lustre/utils/lfs setdirstripe -D -c 2 /mnt/lustre/d33e.sanityn striped subdir creation createmany -d /mnt/lustre/d33e.sanityn/subdir 100 total: 100 mkdir in 0.96 seconds: 104.40 ops/second sub file creation and write createmany -o /mnt/lustre/d33e.sanityn/subfile 100; echo abc > /mnt/lustre/d33e.sanityn/subfile1 total: 100 open/close in 0.99 seconds: 101.09 ops/second sub file append write echo dfg >> /mnt/lustre2/d33e.sanityn/subfile2 subdir setatttr touch /mnt/lustre2/d33e.sanityn/subdir1 subdir unlink unlinkmany -d /mnt/lustre/d33e.sanityn/subdir 100 - unlinked 0 (time 1746157371 ; total 0 ; last 0) total: 100 unlinks in 3 seconds: 33.333332 unlinks/second sub file unlink unlinkmany /mnt/lustre2/d33e.sanityn/subfile 100 - unlinked 0 (time 1746157374 ; total 0 ; last 0) total: 100 unlinks in 2 seconds: 50.000000 unlinks/second sub file creation follows striped dir chmod chmod 777 /mnt/lustre/d33e.sanityn/striped; touch /mnt/lustre/d33e.sanityn/striped/subfile striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn/striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn CoS count 0 PASS 33e (11s) == sanityn test 34: no lock timeout under IO ============= 23:42:59 (1746157379) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg344-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800ae0f0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ae0f0000.ost_server_uuid in FULL state after 0 sec oleg344-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800ae0f0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800ae0f0000.ost_server_uuid in IDLE state after 0 sec oleg344-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800ae0f0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ae0f0000.ost_server_uuid in FULL state after 0 sec oleg344-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800ae0f0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800ae0f0000.ost_server_uuid in IDLE state after 0 sec oleg344-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800ae0f0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ae0f0000.ost_server_uuid in IDLE state after 0 sec oleg344-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800ae0f0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800ae0f0000.ost_server_uuid in IDLE state after 0 sec PASS 34 (87s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 23:44:26 (1746157466) striped dir -i1 -c2 -H all_char /mnt/lustre/d35.sanityn fail_loc=0x80000317 Race attempt 0 Wait for 28630 28769 for 60 sec... - open/close 1793 (time 1746157479.60 total 10.00 last 179.29) - open/close 3386 (time 1746157489.60 total 20.00 last 159.29) total: 4000 open/close in 23.04 seconds: 173.60 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.144@tcp:/lustre 7.4G 64M 6.9G 1% /mnt/lustre 192.168.203.144@tcp:/lustre 7.4G 64M 6.9G 1% /mnt/lustre2 PASS 35 (66s) == sanityn test 36: handle ESTALE/open-unlink correctly == 23:45:32 (1746157532) striped dir -i0 -c2 -H crush2 /mnt/lustre/d36.sanityn Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg344-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.458881 s, 114 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7670 Waiting for MDT destroys to complete *** cycle(0) *** before(7148684) after_dd(7097484) after(7148684) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.469043 s, 112 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7670 Waiting for MDT destroys to complete *** cycle(1) *** before(7148684) after_dd(7097484) after(7148684) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.454011 s, 115 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7670 Waiting for MDT destroys to complete *** cycle(2) *** before(7148684) after_dd(7097484) after(7148684) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.382922 s, 137 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7670 Waiting for MDT destroys to complete *** cycle(3) *** before(7148684) after_dd(7097484) after(7148684) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.506649 s, 103 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7670 Waiting for MDT destroys to complete *** cycle(4) *** before(7148684) after_dd(7097484) after(7148684) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.482866 s, 109 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7670 Waiting for MDT destroys to complete *** cycle(5) *** before(7148684) after_dd(7097484) after(7148684) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.56891 s, 92.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7670 Waiting for MDT destroys to complete *** cycle(6) *** before(7148684) after_dd(7097484) after(7148684) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.386886 s, 136 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7670 Waiting for MDT destroys to complete *** cycle(7) *** before(7148684) after_dd(7097484) after(7148684) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.513122 s, 102 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7670 Waiting for MDT destroys to complete *** cycle(8) *** before(7148684) after_dd(7097484) after(7148684) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.449821 s, 117 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7670 Waiting for MDT destroys to complete *** cycle(9) *** before(7148684) after_dd(7097484) after(7148684) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.510939 s, 103 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7670 Waiting for MDT destroys to complete *** cycle(10) *** before(7148684) after_dd(7097484) after(7148684) PASS 36 (175s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 23:48:27 (1746157707) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d37.sanityn multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7670 - create 3409 (time 1746157719.17 total 10.00 last 340.85) - create 6262 (time 1746157729.17 total 20.00 last 285.28) - create 9203 (time 1746157739.18 total 30.00 last 294.03) total: 10000 create in 32.59 seconds: 306.85 ops/second PASS 37 (37s) == sanityn test 39a: file mtime does not change after rename ========================================================== 23:49:04 (1746157744) repeat after cancel_lru_locks PASS 39a (3s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 23:49:07 (1746157747) repeat after cancel_lru_locks PASS 39b (4s) == sanityn test 39c: check truncate mtime update ================================================================================ 23:49:11 (1746157751) repeat after cancel_lru_locks PASS 39c (4s) == sanityn test 39d: sync write should update mtime ====== 23:49:15 (1746157755) fail_loc=0x411 fail_loc=0 PASS 39d (4s) == sanityn test 40a: pdirops: create vs others ======================================================================== 23:49:19 (1746157759) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40a (10s) == sanityn test 40b: pdirops: open|create and others ======================================================================== 23:49:29 (1746157769) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (10s) == sanityn test 40c: pdirops: link and others ======================================================================== 23:49:39 (1746157779) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (10s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 23:49:49 (1746157789) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (10s) == sanityn test 40e: pdirops: rename and others ======================================================================== 23:50:00 (1746157800) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (11s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 23:50:10 (1746157810) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (8s) == sanityn test 41b: pdirops: create vs create ======================================================================== 23:50:18 (1746157818) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 open: File exists Conflict PASS 41b (7s) == sanityn test 41c: pdirops: create vs link ======================================================================== 23:50:25 (1746157825) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (8s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 23:50:33 (1746157833) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41d (7s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 23:50:41 (1746157841) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41e (8s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 23:50:49 (1746157849) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41f (8s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 23:50:56 (1746157856) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41g (8s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 23:51:04 (1746157864) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 41h (7s) == sanityn test 41i: reint_open: create vs create ======== 23:51:11 (1746157871) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (758s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 00:03:49 (1746158629) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (7s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 00:03:56 (1746158636) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 open: File exists Conflict PASS 42b (7s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 00:04:03 (1746158643) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (6s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 00:04:09 (1746158649) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 Conflict PASS 42d (6s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 00:04:15 (1746158655) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (6s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 00:04:21 (1746158661) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 Conflict PASS 42f (7s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 00:04:28 (1746158668) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 42g (6s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 00:04:34 (1746158674) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 Conflict PASS 42h (7s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 00:04:41 (1746158681) PASS 43a (53s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 00:05:34 (1746158734) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 Conflict PASS 43b (7s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 00:05:41 (1746158741) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 Conflict PASS 43c (7s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 00:05:48 (1746158748) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (6s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 00:05:54 (1746158754) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43e (5s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 00:05:59 (1746158759) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (6s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 00:06:05 (1746158765) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 stat: cannot stat '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (6s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 00:06:11 (1746158771) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43h (6s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 00:06:17 (1746158777) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43i (7s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 00:06:24 (1746158784) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (59s) == sanityn test 43k: unlink vs create ==================== 00:07:23 (1746158843) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (753s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 00:19:56 (1746159596) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri May 2 00:20:01 EDT 2025 PASS 44a (8s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 00:20:04 (1746159604) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 44b (7s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 00:20:11 (1746159611) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (7s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 00:20:18 (1746159618) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 Conflict PASS 44d (7s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 00:20:26 (1746159626) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 Conflict PASS 44e (5s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 00:20:31 (1746159631) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=8 Conflict PASS 44f (7s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 00:20:38 (1746159638) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=8 Conflict PASS 44g (6s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 00:20:45 (1746159645) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 Conflict PASS 44h (6s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 00:20:51 (1746159651) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 lfs mkdir: dirstripe error on '/mnt/lustre2/f44i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f44i.sanityn': File exists Conflict PASS 44i (7s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 00:20:58 (1746159658) PASS 45a (91s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 00:22:29 (1746159749) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45b (9s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 00:22:38 (1746159758) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45c (9s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 00:22:47 (1746159767) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (8s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 00:22:55 (1746159775) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45e (8s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 00:23:04 (1746159784) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (9s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 00:23:12 (1746159792) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 stat: cannot stat '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (8s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 00:23:20 (1746159800) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45h (8s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 00:23:28 (1746159808) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45i (8s) == sanityn test 45j: read vs rename ====================== 00:23:36 (1746159816) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (709s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 00:35:26 (1746160526) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (8s) == sanityn test 46b: pdirops: link vs create ======================================================================== 00:35:34 (1746160534) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 46b (8s) == sanityn test 46c: pdirops: link vs link ======================================================================== 00:35:43 (1746160543) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (8s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 00:35:51 (1746160551) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 Conflict PASS 46d (7s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 00:35:58 (1746160558) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46e (8s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 00:36:06 (1746160566) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46f (8s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 00:36:14 (1746160574) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46g (8s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 00:36:22 (1746160582) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46h (8s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 00:36:30 (1746160590) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 lfs mkdir: dirstripe error on '/mnt/lustre2/f46i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f46i.sanityn': File exists Conflict PASS 46i (9s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 00:36:39 (1746160599) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f47a.sanityn': File exists Conflict PASS 47a (7s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 00:36:46 (1746160606) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 open: File exists Conflict PASS 47b (9s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 00:36:55 (1746160615) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f47c.sanityn' to '/mnt/lustre2/f47c.sanityn-2': File exists Conflict PASS 47c (7s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 00:37:02 (1746160622) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 47d (7s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 00:37:09 (1746160629) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f47e.sanityn' with non-directory Conflict PASS 47e (8s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 00:37:17 (1746160637) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 47f (8s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 00:37:25 (1746160645) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 47g (8s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 00:37:33 (1746160653) 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.0104327 s, 982 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (8s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 00:37:41 (1746160661) 0+1 records in 0+1 records out 159 bytes (159 B) copied, 0.0219551 s, 7.2 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3519: kill: (3785) - No such process PASS 51a (6s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 00:37:47 (1746160667) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0110747 s, 92.5 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.032148 s, 32.6 MB/s 3145728 PASS 51b (15s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 00:38:02 (1746160682) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0173112 s, 59.2 kB/s PASS 51c (7s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 00:38:09 (1746160689) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0252198 s, 41.6 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (4s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 00:38:13 (1746160693) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 1299 0x513 0x2c0000401 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 1299 0x513 0x2c0000401 read lease(1) has applied. PASS 51e (6s) == sanityn test 54: rename locking ======================= 00:38:19 (1746160699) llite.lustre-ffff8800ae0f0000.xattr_cache=1 llite.lustre-ffff8800b60dc000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: '/mnt/lustre/d1/d2' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762666 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-02 00:38:20.000000000 -0400 Modify: 2025-05-02 00:38:20.000000000 -0400 Change: 2025-05-02 00:38:20.000000000 -0400 Birth: - ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: '/mnt/lustre/d1/d2' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281277 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-02 00:38:26.000000000 -0400 Modify: 2025-05-02 00:38:26.000000000 -0400 Change: 2025-05-02 00:38:26.000000000 -0400 Birth: - ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: '/mnt/lustre/d1/d2' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762669 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-02 00:38:31.000000000 -0400 Modify: 2025-05-02 00:38:31.000000000 -0400 Change: 2025-05-02 00:38:31.000000000 -0400 Birth: - open(O_RDWR|O_CREAT): Is a directory File: '/mnt/lustre/d1' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281278 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-02 00:38:31.000000000 -0400 Modify: 2025-05-02 00:38:32.000000000 -0400 Change: 2025-05-02 00:38:32.000000000 -0400 Birth: - ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: '/mnt/lustre/d1/d2' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281280 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-02 00:38:37.000000000 -0400 Modify: 2025-05-02 00:38:37.000000000 -0400 Change: 2025-05-02 00:38:37.000000000 -0400 Birth: - File: '/mnt/lustre/d1' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762670 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-02 00:38:37.000000000 -0400 Modify: 2025-05-02 00:38:37.000000000 -0400 Change: 2025-05-02 00:38:37.000000000 -0400 Birth: - PASS 54 (26s) == sanityn test 55a: rename vs unlink target dir ========= 00:38:45 (1746160725) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (8s) == sanityn test 55b: rename vs unlink source dir ========= 00:38:53 (1746160733) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55b.sanityn/d1/d2' to '/mnt/lustre/d55b.sanityn/d3/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 00:39:01 (1746160741) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (14s) == sanityn test 55d: rename file vs link ================= 00:39:15 (1746160755) fail_loc=0x155 mv: cannot move '/mnt/lustre/d55d.sanityn/f1' to '/mnt/lustre/d55d.sanityn/d55d.sanityn': Is a directory PASS 55d (11s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 00:39:26 (1746160766) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1746160767 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 12.2078 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (9s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 00:39:35 (1746160775) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: read /mnt/lustre/f56b.sanityn@33554431995904+4096 no data: Success llverdev: read /mnt/lustre/f56b.sanityn@33554431995904+4096 no data: Success llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1746160777 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.59042 MB/s read offset: 32768000000kB -nan MB/s llverdev: data verified successfully PASS 56b (41s) == sanityn test 60: Verify data_version behaviour ======== 00:40:16 (1746160816) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60.sanityn PASS 60 (4s) == sanityn test 70a: cd directory && rm directory ======== 00:40:20 (1746160820) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 00:40:23 (1746160823) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 00:40:27 (1746160827) 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.0042943 s, 15.3 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00889476 s, 7.4 MB/s File: '/mnt/lustre2/f71a.sanityn' Size: 262144 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205289281303 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-02 00:40:27.000000000 -0400 Modify: 2025-05-02 00:40:27.000000000 -0400 Change: 2025-05-02 00:40:27.000000000 -0400 Birth: - 27916 extent 0 in offset 65536, length 65536 flags (0x80000000): NET extent 1 in offset 196608, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 2, sum of lengths 131072, expected sum 131072 27917 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00440276 s, 14.9 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00784898 s, 8.3 MB/s File: '/mnt/lustre/f71a.sanityn' Size: 262144 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205289281303 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-02 00:40:27.000000000 -0400 Modify: 2025-05-02 00:40:27.000000000 -0400 Change: 2025-05-02 00:40:27.000000000 -0400 Birth: - 27918 extent 0 in offset 65536, length 65536 flags (0x80000000): NET extent 1 in offset 196608, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 2, sum of lengths 131072, expected sum 131072 27917 PASS 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 00:40:30 (1746160830) 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.0106258 s, 6.2 MB/s extent 0 in offset 0, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 1, sum of lengths 65536, expected sum 65536 PASS 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 00:40:33 (1746160833) filefrag 1.46.2.wc5 (26-Mar-2022) check last flag for file with 291 extents 290: 296960.. 296963: 3508224.. 3508227: 4: 0001: last,net,eof check last flag for file with 292 extents 291: 297984.. 297987: 855040.. 855043: 4: 0000: last,net,eof check last flag for file with 293 extents 292: 299008.. 299011: 3911680.. 3911683: 4: 0001: last,net,eof PASS 71c (21s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 00:40:54 (1746160854) 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.014432 s, 4.5 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00475489 s, 13.8 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00250582 s, 26.2 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.0387558 s, 1.7 MB/s disk usage: 262144 /mnt/lustre/f71d.sanityn file size: 458752 /mnt/lustre/f71d.sanityn extent 0 in offset 0, length 65536 flags (0x80000000): NET extent 1 in offset 131072, length 65536 flags (0x80000000): NET extent 2 in offset 262144, length 65536 flags (0x80000000): NET extent 3 in offset 393216, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 4, sum of lengths 262144, expected sum 262144 PASS 71d (21s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 00:41:15 (1746160875) llite.lustre-ffff8800ae0f0000.xattr_cache=1 llite.lustre-ffff8800b60dc000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (4s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 00:41:19 (1746160879) llite.lustre-ffff8800ae0f0000.xattr_cache=1 llite.lustre-ffff8800b60dc000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (3s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 00:41:22 (1746160882) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00815236 s, 126 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0131403 s, 77.9 kB/s 18802: get lock2 18803: get lock1 18802: done 18802: get lock1 18803: done 18803: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 18803: cannot get lock2: Resource deadlock avoided 18803: exit rc=0 18802: done 18802: exit rc=0 PASS 74 (7s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 00:41:29 (1746160889) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0234066 s, 89.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0228848 s, 45.8 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4108: 19453 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (9s) == sanityn test 76: Verify MDT open_files listing ======== 00:41:38 (1746160898) striped dir -i0 -c2 -H all_char /mnt/lustre/d76.sanityn opening files: ................ closing files: ................ PASS 76 (78s) == sanityn test 77a: check FIFO NRS policy =============== 00:42:56 (1746160976) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.191921 s, 87.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0497204 s, 21.1 MB/s , 0.0196869 s, 53.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0462322 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0509042 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0918683 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.100688 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0802917 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0623199 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.108191 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.11831 s, 8.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.09496 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.123236 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.115775 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.126816 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.124664 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.12344 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0468102 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0385607 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00318219 s, 330 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0019272 s, 544 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.061569 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0754308 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0721061 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00479984 s, 218 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.068172 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0608039 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00602155 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0406841 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0353865 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0324721 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.028707 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0228745 s, 45.8 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg344-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77a.sanityn' PASS 77a (5s) == sanityn test 77b: check CRR-N NRS policy ============== 00:43:01 (1746160981) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.244915 s, 68.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0449726 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0517921 s, 20.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0943946 s, 11.1 MB/s 1048576 bytes (1.0 MB) copied, 0.090626 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.112544 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.102472 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.106962 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.112993 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0870693 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0824301 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.10506 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.117564 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.109645 s, 9.6 MB/s 1048576 bytes (1.0 MB) copied, 0.0982345 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.106431 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.13954 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0648703 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0470031 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0683321 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0480144 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0498012 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0464856 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0488305 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0537267 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0530753 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0559052 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0590386 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0575461 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0668931 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0671573 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0683338 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0872975 s, 12.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=2 removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg344-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.224009 s, 74.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0400555 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0449481 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0705325 s, 14.9 MB/s , 0.0625033 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0882203 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.11019 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.1165 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.119322 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.107938 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.115853 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.12457 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.12134 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.130538 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.12361 s, 8.5 MB/s 1048576 bytes (1.0 MB) copied, 0.129001 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.124265 s, 8.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0636013 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.049589 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0506353 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0549355 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0660779 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0642467 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.063659 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0701433 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0700349 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0625964 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0759229 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.065161 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.074819 s, 14.0 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0674534 s, 15.5 MB/s , 0.075647 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0751226 s, 14.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=8 removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg344-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (8s) == sanityn test 77c: check ORR NRS policy ================ 00:43:09 (1746160989) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.254204 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0470227 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0587441 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0721475 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0632463 s, 16.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0464786 s, 22.6 MB/s 1048576 bytes (1.0 MB) copied, 0.0842031 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0925987 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.10371 s, 10.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0955313 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0618948 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0835808 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.101429 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.130106 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.131122 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0857237 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.114386 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0314898 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0241515 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00305631 s, 343 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0214229 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00169522 s, 619 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0277596 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0661998 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00353038 s, 297 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0021166 s, 495 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0409928 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0491178 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0274949 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0305547 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0428787 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0420695 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0446652 s, 23.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg344-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.21042 s, 79.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0180351 s, 58.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0178377 s, 58.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0615411 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0577455 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0521457 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0651641 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0621019 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0537518 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0775417 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0920504 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.090445 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0754693 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0990625 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.105879 s, 9.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.111318 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.106275 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00279231 s, 376 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00181005 s, 579 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0038982 s, 269 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0950605 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00208131 s, 504 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0650619 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.068793 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0828467 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0609198 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.11203 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.121886 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0269116 s, 39.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0516228 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0212795 s, 49.3 MB/s , 0.032755 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0474469 s, 22.1 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg344-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.196805 s, 85.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0287296 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.024805 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.048014 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0933999 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.107868 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.110222 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0894971 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.118598 s, 8.8 MB/s 1048576 bytes (1.0 MB) copied, 0.0998887 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.126589 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.10343 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.131552 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.119064 s, 8.8 MB/s 1048576 bytes (1.0 MB) copied, 0.132201 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.142514 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.157543 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00225599 s, 465 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0235209 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0763256 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0452978 s, 23.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0310544 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0017636 s, 595 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00340704 s, 308 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0291369 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0044865 s, 234 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00178185 s, 588 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00206371 s, 508 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0527046 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00169423 s, 619 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0520843 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0471099 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00923447 s, 114 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg344-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (7s) == sanityn test 77d: check TRR nrs policy ================ 00:43:16 (1746160996) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.184392 s, 91.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0256338 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.025892 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0451955 s, 23.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.121255 s, 8.6 MB/s , 0.111647 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.120287 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0928116 s, 11.3 MB/s , 0.0956414 s, 11.0 MB/s 1048576 bytes (1.0 MB) copied, 0.121191 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.127088 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.115485 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.109587 s, 9.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.107524 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.106244 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.134761 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.125034 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0351968 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0584847 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0095322 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0346862 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00184579 s, 568 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00381665 s, 275 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0881503 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0838857 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00456368 s, 230 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0827362 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0624733 s, 16.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0176944 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0403703 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00191469 s, 548 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0142185 s, 73.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00908575 s, 115 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg344-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.164885 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117222 s, 89.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0332363 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0319158 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.042915 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0321797 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0636817 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0476978 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0554504 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0807756 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0935316 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0986224 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0777201 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0779881 s, 13.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0882073 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.100872 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0863409 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0611194 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.075452 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0514981 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00180904 s, 580 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00194995 s, 538 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00198477 s, 528 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0788122 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0688483 s, 15.2 MB/s , 0.0791783 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0869933 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0814151 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0530092 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.044782 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.031497 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.017756 s, 59.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.023896 s, 43.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=2 removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg344-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.181525 s, 92.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0179762 s, 58.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0325757 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0410132 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.075147 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0529586 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0849183 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0874066 s, 12.0 MB/s , 0.0767401 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0689129 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0939438 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.127935 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.111467 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.139031 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.149792 s, 7.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.180678 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.211181 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00234748 s, 447 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0365523 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00247544 s, 424 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0976079 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.115788 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0995344 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0800328 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0810965 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0821586 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0653416 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0567098 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.078172 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0966324 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0648093 s, 16.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0286334 s, 36.6 MB/s , 0.0239163 s, 43.8 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg344-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (8s) == sanityn test 77e: check TBF NID nrs policy ============ 00:43:24 (1746161004) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.44@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.233873 s, 71.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0227676 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0426636 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0778295 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0923855 s, 11.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.111467 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.13659 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.140584 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.152736 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.148927 s, 7.0 MB/s 1048576 bytes (1.0 MB) copied, 0.159926 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.176653 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.174843 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.17816 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.181112 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.21887 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.235158 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0449351 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0866524 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0898359 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.102986 s, 10.2 MB/s 1048576 bytes (1.0 MB) copied, 0.038759 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0409339 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0366115 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0400888 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0408622 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.043172 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0438197 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0438838 s, 23.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0492569 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0505234 s, 20.8 MB/s 1048576 bytes (1.0 MB) copied, 0.0509834 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0609015 s, 17.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=3 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg344-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.180576 s, 92.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0361114 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0350426 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0230541 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0392531 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0834669 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0627406 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0509438 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.055062 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0715552 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0849599 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0847385 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0902883 s, 11.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.13755 s, 7.6 MB/s , 0.115727 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0977751 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.135356 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0518552 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00182513 s, 575 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00199232 s, 526 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0975443 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0873029 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.105515 s, 9.9 MB/s , 0.110253 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0779516 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.101298 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.115692 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0992883 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0741995 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.108906 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0736751 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.041771 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.035257 s, 29.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=9 removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg344-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.182841 s, 91.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0161327 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0239012 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0275461 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0412747 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0386811 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0625177 s, 16.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0702489 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0808649 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.11344 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0843431 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0895261 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.126352 s, 8.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0966717 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.106076 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.113252 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.103058 s, 10.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0499528 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0723697 s, 14.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0572774 s, 18.3 MB/s 1048576 bytes (1.0 MB) copied, 0.043748 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0445905 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.04809 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0210159 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0523136 s, 20.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0522492 s, 20.1 MB/s 1048576 bytes (1.0 MB) copied, 0.0790954 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0571326 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0467556 s, 22.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0424462 s, 24.7 MB/s , 0.0431311 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0677261 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0455907 s, 23.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=1 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg344-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.236905 s, 70.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0435985 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0439009 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0666565 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0695313 s, 15.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.13024 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.110119 s, 9.5 MB/s 1048576 bytes (1.0 MB) copied, 0.129299 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.110323 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.117141 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.134573 s, 7.8 MB/s 1048576 bytes (1.0 MB) copied, 0.14683 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.15396 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.142001 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.137397 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.149782 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.168484 s, 6.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0015691 s, 668 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0396168 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0188349 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00227295 s, 461 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0407838 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0384775 s, 27.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00193909 s, 541 MB/s , 0.00915544 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0417883 s, 25.1 MB/s 1048576 bytes (1.0 MB) copied, 0.00747305 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0609291 s, 17.2 MB/s , 0.0438171 s, 23.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0455911 s, 23.0 MB/s , 0.0423194 s, 24.8 MB/s 1048576 bytes (1.0 MB) copied, 0.0388286 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0442714 s, 23.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg344-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' PASS 77e (14s) == sanityn test 77f: check TBF JobID nrs policy ========== 00:43:38 (1746161018) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg344-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.229788 s, 73.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0155724 s, 67.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0156931 s, 66.8 MB/s 1048576 bytes (1.0 MB) copied, 0.0197624 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0268925 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0262592 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0135232 s, 77.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0287411 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0486517 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0407741 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0208335 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0247039 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.164254 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.181229 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.198201 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.188452 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.191444 s, 5.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0180352 s, 58.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00195611 s, 536 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00847531 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00197982 s, 530 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00166241 s, 631 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0700563 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0768707 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0474291 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0565552 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0027642 s, 379 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00507548 s, 207 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0439816 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0156334 s, 67.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0137944 s, 76.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0610959 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011134 s, 94.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg344-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg344-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.195949 s, 85.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0256497 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0201014 s, 52.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0309687 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0457121 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0493609 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0609163 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0626581 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0589582 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0375285 s, 27.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0344413 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0646995 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0520929 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0691355 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0853447 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0644804 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0805437 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0246041 s, 42.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0325223 s, 32.2 MB/s 1048576 bytes (1.0 MB) copied, 0.00204052 s, 514 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00181219 s, 579 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0016595 s, 632 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.073787 s, 14.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0541479 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.060497 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00189809 s, 552 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0475013 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0338644 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0228674 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0198656 s, 52.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.01065 s, 98.5 MB/s , 0.0132515 s, 79.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0174542 s, 60.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg344-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg344-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.216157 s, 77.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0290122 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0299098 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0778319 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0792291 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0390846 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.070972 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0577796 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0980054 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0681641 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0930834 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0689542 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.101237 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0933937 s, 11.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.102415 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.110551 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.102201 s, 10.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00183678 s, 571 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00328102 s, 320 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0582432 s, 18.0 MB/s , 0.0355436 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0716548 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0430821 s, 24.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0522038 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0522573 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0382648 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.033426 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0302862 s, 34.6 MB/s 1048576 bytes (1.0 MB) copied, 0.0185658 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00410693 s, 255 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0159939 s, 65.6 MB/s , 0.0788984 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0284578 s, 36.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=5 removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg344-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg344-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.194294 s, 86.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.019184 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0150682 s, 69.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0400871 s, 26.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0193394 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0447121 s, 23.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.110229 s, 9.5 MB/s 1048576 bytes (1.0 MB) copied, 0.113522 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.107815 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.123427 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.114171 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.126018 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.129565 s, 8.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.139214 s, 7.5 MB/s 1048576 bytes (1.0 MB) copied, 0.139962 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.150809 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.159665 s, 6.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00180313 s, 582 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00195546 s, 536 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00174696 s, 600 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00163149 s, 643 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.145385 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.129978 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.11754 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.108962 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.10712 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108425 s, 96.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0596189 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0386757 s, 27.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.039019 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0289207 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0237934 s, 44.1 MB/s 1048576 bytes (1.0 MB) copied, 0.114319 s, 9.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg344-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' PASS 77f (12s) == sanityn test 77g: Change TBF type directly ============ 00:43:50 (1746161030) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (4s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 00:43:55 (1746161035) oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg344-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg344-client: oleg344-server: ssh exited with exit code 2 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Unknown error 524 oleg344-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Unknown error 524 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 12 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Unknown error 524 oleg344-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Unknown error 524 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 12 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Unknown error 524 oleg344-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Unknown error 524 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 12 PASS 77h (5s) == sanityn test 77i: Change rank of TBF rule ============= 00:44:00 (1746161040) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (11s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 00:44:11 (1746161051) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.216833 s, 77.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0215331 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0316016 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0628566 s, 16.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0827014 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0654738 s, 16.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0842215 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0851604 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0939653 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0736163 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0947322 s, 11.1 MB/s , 0.0791702 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0981292 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0801743 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0878811 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.103504 s, 10.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.116866 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0211883 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.014961 s, 70.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00158673 s, 661 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.028965 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0185409 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0161803 s, 64.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124308 s, 84.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0229605 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.002231 s, 470 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0246219 s, 42.6 MB/s , 0.0311744 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0254553 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0321042 s, 32.7 MB/s , 0.0263505 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0291214 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0302739 s, 34.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=1 removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg344-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.96525 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.4845 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (42s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 00:44:54 (1746161094) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg344-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg344-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg344-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg344-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg344-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.322557 s, 52.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0408853 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.057088 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0609279 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.10031 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.105769 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.107009 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.121743 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.113024 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.105437 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.112504 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.117467 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0416277 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.158644 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.161435 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.165468 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0694917 s, 15.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0214589 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0287815 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0395102 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0359923 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0287555 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0305676 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0423701 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0457364 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.031777 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0365642 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0388263 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0400192 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0439236 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0457207 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0479241 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0555808 s, 18.9 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg344-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.7249 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9347 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg344-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg344-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg344-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg344-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg344-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.302368 s, 55.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0323617 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0385753 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0482206 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0510721 s, 20.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.064405 s, 16.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0612772 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0790958 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.080702 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0898981 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0825811 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0958305 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0990622 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.139911 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.174597 s, 6.0 MB/s 1048576 bytes (1.0 MB) copied, 0.168811 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.178991 s, 5.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0489345 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0334899 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0499615 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0399093 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0452548 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0657806 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0503198 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.070174 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0527835 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0728671 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0474308 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0751505 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0507995 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0514949 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0530251 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0597563 s, 17.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg344-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.7636 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9264 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (116s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 00:46:49 (1746161209) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg344-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg344-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg344-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg344-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg344-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.30929 s, 54.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0364175 s, 28.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0514494 s, 20.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0479835 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0574381 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0790578 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0790156 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115586 s, 90.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0206932 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0738632 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.111459 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.094167 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.112806 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0984875 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.144253 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.123928 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.16229 s, 6.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0318176 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0345987 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0306375 s, 34.2 MB/s , 0.0307038 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0319922 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0322053 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0335449 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0376796 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0379145 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.040465 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0375119 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0399175 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0411991 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0429418 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0409337 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0484544 s, 21.6 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg344-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.7114 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9399 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg344-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg344-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg344-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg344-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg344-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.310153 s, 54.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0802575 s, 13.1 MB/s 1048576 bytes (1.0 MB) copied, 0.0390571 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0648236 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0691425 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0646025 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0742735 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0684225 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0753358 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0728259 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.071069 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0759644 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.130162 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.132908 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.133507 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.125015 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.143306 s, 7.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.018775 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.020625 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0215242 s, 48.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0246106 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0265239 s, 39.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0409966 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0335039 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0345801 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0396287 s, 26.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0375546 s, 27.9 MB/s , 0.0450267 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0465313 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0441437 s, 23.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0525182 s, 20.0 MB/s , 0.0408654 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0584379 s, 17.9 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg344-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.7103 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9985 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (114s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 00:48:44 (1746161324) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg344-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.209709 s, 80.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0268737 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0388239 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.046743 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0554637 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0534901 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0727318 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0583359 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0781387 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0985675 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.080613 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0927899 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.112701 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0980378 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0906128 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.10306 s, 10.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.105803 s, 9.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.038181 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0516641 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0508027 s, 20.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.051739 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0541555 s, 19.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0557577 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0437834 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0440022 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0407203 s, 25.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0287849 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0428523 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0178973 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0238496 s, 44.0 MB/s , 0.04374 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0517634 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0249295 s, 42.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=5 removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg344-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.97111 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.81525 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.44@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.44@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.147206 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0278812 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0308081 s, 34.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0530626 s, 19.8 MB/s 1048576 bytes (1.0 MB) copied, 0.056422 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0592201 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0540209 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0680849 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0578963 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0695995 s, 15.1 MB/s , 0.0661058 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0622291 s, 16.9 MB/s , 0.0732826 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0610782 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0534922 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0708189 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0726971 s, 14.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0743386 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0682014 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0755296 s, 13.9 MB/s , 0.073703 s, 14.2 MB/s , 0.0738973 s, 14.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0787148 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0793791 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0730095 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0735252 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0824854 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0825183 s, 12.7 MB/s , 0.0831317 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0736391 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0739229 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0761149 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0779226 s, 13.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg344-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.94868 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.80251 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.44@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg344-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.134696 s, 125 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104726 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118553 s, 88.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0201892 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0262682 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0289544 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0356676 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.069087 s, 15.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0737315 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0593522 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0654907 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0690744 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.08484 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0823539 s, 12.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.102955 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.106548 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.101419 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125888 s, 83.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0207719 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01649 s, 63.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00438224 s, 239 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102133 s, 103 MB/s 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0128003 s, 81.9 MB/s , 0.00772014 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00165423 s, 634 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00159941 s, 656 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00174596 s, 601 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0342575 s, 30.6 MB/s , 0.0194202 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00252765 s, 415 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01608 s, 65.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132546 s, 79.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00988195 s, 106 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg344-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.95403 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.92224 s, 21.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg344-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.228524 s, 73.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0204184 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.026726 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0650099 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0697351 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0711018 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0714507 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0760034 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0908662 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0754357 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0873236 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0811673 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0916404 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0824196 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0815758 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0971903 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.123042 s, 8.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=3 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00795843 s, 132 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00168509 s, 622 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0726905 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0546395 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0626875 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0449235 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0567461 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0546779 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0439613 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0463336 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0418135 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0394679 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0354323 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.04339 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0239181 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0357649 s, 29.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg344-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.84185 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.79728 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.98952 s, 21.0 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.83065 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg344-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.16991 s, 98.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0170241 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0187771 s, 55.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.051082 s, 20.5 MB/s , 0.0455352 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0555055 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0465591 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0561095 s, 18.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0605731 s, 17.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0701031 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0756192 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0718785 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0704261 s, 14.9 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.077728 s, 13.5 MB/s , 0.0775059 s, 13.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0785858 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0796044 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0216033 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0259127 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0221901 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0293707 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0211015 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0227033 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0247964 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.025212 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0279447 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0285319 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0263442 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0282992 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0296323 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0267947 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0245731 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0338234 s, 31.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg344-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.5023 s, 5.4 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.5127 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg344-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.195147 s, 86.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0284762 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0322548 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0668799 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0634772 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.085434 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0915819 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.10469 s, 10.0 MB/s 1048576 bytes (1.0 MB) copied, 0.112526 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.113216 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.115183 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.107776 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.119863 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.12726 s, 8.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.12827 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.126752 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.147066 s, 7.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00301961 s, 347 MB/s 1048576 bytes (1.0 MB) copied, 0.0060428 s, 174 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00191553 s, 547 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00420069 s, 250 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0055101 s, 190 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00191745 s, 547 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00175262 s, 598 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.110054 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.100322 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.134303 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.127204 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0931475 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0733818 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0911685 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0270839 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0883712 s, 11.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg344-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.98419 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.7894 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg344-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.192118 s, 87.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0206338 s, 50.8 MB/s 1048576 bytes (1.0 MB) copied, 0.03367 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.024705 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0352695 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0784064 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0668826 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0980519 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.114654 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.116641 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.108274 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.120689 s, 8.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.132693 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.136519 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.142613 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.146811 s, 7.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.16247 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0219994 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0375201 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0387507 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.018806 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0408157 s, 25.7 MB/s , 0.0314993 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0207939 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0339644 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0221069 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0237917 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.023937 s, 43.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0307971 s, 34.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0311115 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0277308 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0302232 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0354375 s, 29.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=7 removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg344-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.82632 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.79653 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.78929 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.77469 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (271s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 00:53:14 (1746161594) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (4s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 00:53:18 (1746161598) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg344-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 3.72447 s, 282 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg344-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 40.0174 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (51s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 00:54:09 (1746161649) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.152423 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0175369 s, 59.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01958 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0274395 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.045284 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0417952 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0463707 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0376564 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0379456 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0561191 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0528102 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0551343 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0529161 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0461372 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0556156 s, 18.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0602234 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0675954 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101072 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00234072 s, 448 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.040911 s, 25.6 MB/s , 0.0240412 s, 43.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0382498 s, 27.4 MB/s , 0.0349075 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0477699 s, 22.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0163093 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.053541 s, 19.6 MB/s 1048576 bytes (1.0 MB) copied, 0.0184713 s, 56.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0144801 s, 72.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121638 s, 86.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0043475 s, 241 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00819606 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107665 s, 97.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00765136 s, 137 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg344-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.92678 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 5.02201 s, 20.9 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.151361 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00987004 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133928 s, 78.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0135338 s, 77.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0365216 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0359693 s, 29.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0524442 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0382087 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0400717 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0616057 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0579057 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.077408 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0658606 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0631985 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0765354 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0896611 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0894762 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0272175 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00812326 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132085 s, 79.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108716 s, 96.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00238876 s, 439 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0411247 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0307198 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0229985 s, 45.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0220043 s, 47.7 MB/s , 0.0190481 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117528 s, 89.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.016463 s, 63.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0147942 s, 70.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00341558 s, 307 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00976408 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00754519 s, 139 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800ae0f0000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8800ae0f0000.lock_unused_count=7 removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg344-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg344-client.virtnet' removed directory: '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.97257 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.97725 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (43s) == sanityn test 77o: Changing rank should not panic ====== 00:54:53 (1746161693) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 00:54:58 (1746161698) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (43s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 00:55:41 (1746161741) mds.MDS.mdt.nrs_policies=tbf oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg344-client: oleg344-server: ssh exited with exit code 36 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 oleg344-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg344-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg344-client: oleg344-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (14s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 00:55:55 (1746161755) striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d77r.sanityn mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 17837 kill md_thread xargs PASS 77r (39s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 00:56:35 (1746161795) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 00:56:38 (1746161798) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d79.sanityn File: '/mnt/lustre/d79.sanityn' Size: 8192 Blocks: 16 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798763882 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-02 00:56:39.000000000 -0400 Modify: 2025-05-02 00:56:39.000000000 -0400 Change: 2025-05-02 00:56:39.000000000 -0400 Birth: - fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 00:56:51 (1746161811) total: 10 open/close in 0.11 seconds: 92.31 ops/second multiop /mnt/lustre2/d80a.sanityn/f80a.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7670 Finish migration, then checking.. PASS 80a (5s) == sanityn test 80b: Accessing directory during migration ========================================================== 00:56:56 (1746161816) start migration thread 23888 accessing the migrating directory for 1 minute... concurrent access 227 failures, 808 successes /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7567: 23888 Killed ( while true; do mdt_idx=$((RANDOM % MDSCOUNT)); $LFS migrate -m $mdt_idx $migdir1 &>/dev/null; done ) (wd: ~) PASS 80b (130s) == sanityn test 81a: rename and stat under striped directory ========================================================== 00:59:06 (1746161946) stat: cannot stat 'd0/0': No such file or directory File: 'd0/0' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289282446 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-02 00:59:07.000000000 -0400 Modify: 2025-05-02 00:59:07.000000000 -0400 Change: 2025-05-02 00:59:07.000000000 -0400 Birth: - PASS 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 00:59:09 (1746161949) - open/close 1669 (time 1746161961.12 total 10.01 last 166.81) total: 2500 open/close in 14.98 seconds: 166.85 ops/second /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 26648 Killed ( while true; do touch $DIR1/$tdir; done ) (wd: ~) PASS 81b (90s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 01:00:39 (1746162039) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 01:00:40 (1746162040) - open/close 2196 (time 1746162052.52 total 10.00 last 219.56) total: 2500 open/close in 11.36 seconds: 220.14 ops/second - open/close 2021 (time 1746162065.11 total 10.00 last 202.05) total: 2500 open/close in 12.37 seconds: 202.07 ops/second snapshot_time 1746162115.137021780 secs.nsecs start_time 1746162041.127670473 secs.nsecs elapsed_time 74.009351307 secs.nsecs open 2505 samples [usecs] 46 4053 1080085 613240223 close 2504 samples [usecs] 42 4767 334890 122072112 mknod 2501 samples [usecs] 133 3250 903583 417872077 unlink 2500 samples [usecs] 173 18759 1501908 3874189308 mkdir 2 samples [usecs] 537 2311 2848 5629090 rmdir 2 samples [usecs] 2729 2944 5673 16114577 rename 2500 samples [usecs] 183 19334 6188783 31516926077 getattr 3765 samples [usecs] 0 3860 104829 38552909 statfs 15 samples [usecs] 8 20 173 2109 parallel_rename_file 2500 samples [usecs] 183 19334 6188783 31516926077 crossdir_rename 2500 samples [usecs] 183 19334 6188783 31516926077 snapshot_time 1746162115.137219790 secs.nsecs start_time 1746162041.127816496 secs.nsecs elapsed_time 74.009403294 secs.nsecs open 2504 samples [usecs] 45 4920 1183363 772629327 close 2504 samples [usecs] 44 3042 359068 117012484 mknod 2500 samples [usecs] 180 4846 979066 533017426 unlink 2500 samples [usecs] 196 23592 1517076 3857881870 mkdir 2 samples [usecs] 535 7796 8331 61063841 rmdir 2 samples [usecs] 2351 11866 14217 146329157 rename 2500 samples [usecs] 231 14060 5464171 22404166955 getattr 3757 samples [usecs] 5 3087 110057 33168181 statfs 15 samples [usecs] 8 338 568 125578 parallel_rename_file 2500 samples [usecs] 231 14060 5464171 22404166955 crossdir_rename 2500 samples [usecs] 231 14060 5464171 22404166955 PASS 81d (77s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 01:01:57 (1746162117) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7670 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7670 PASS 82 (2s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 01:01:59 (1746162119) start pid 9311 to create/unlink striped directory start pid 9312 to stat striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5573: 9311 Terminated ( cd $DIR1; while true; do $LFS mkdir -i1 -c2 $tdir; rmdir $tdir; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5573: 9312 Terminated ( cd $DIR2; while true; do stat $tdir > /dev/null 2>&1; done ) PASS 83 (124s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 01:04:03 (1746162243) fail_loc=0x8000060b PASS 84 (8s) == sanityn test 85: Lustre API root cache race =========== 01:04:11 (1746162251) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1746162252 Finishing test test1 at 1746162253 PASS 85 (4s) == sanityn test 90: open/create and unlink striped directory ========================================================== 01:04:15 (1746162255) start pid 18968 to open/create under striped directory start pid 18970 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5640: 18968 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; touch $tdir/f{0..3} > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5640: 18970 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 90 (183s) == sanityn test 91: chmod and unlink striped directory === 01:07:18 (1746162438) start pid 26565 to chmod striped directory start pid 26566 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5677: 26565 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; chmod go+w $tdir > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5677: 26566 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 91 (183s) == sanityn test 92: create remote directory under orphan directory ========================================================== 01:10:21 (1746162621) fail_loc=0x1408 mkdir: cannot create directory '/mnt/lustre2/d92.sanityn/dir': No such file or directory lfs setdirstripe: cannot resolve path '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory (2) lfs setdirstripe: '/mnt/lustre2/d92.sanityn/remote_dir' is not on a Lustre filesystem: No such file or directory (2) lfs setdirstripe: cannot create dir '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory fail_loc=0 PASS 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 01:10:25 (1746162625) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4044 0xfcc 0x280000400 1 4037 0xfc5 0x2c0000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5677 0x162d 0x2c0000401 0 5717 0x1655 0x280000401 rc1=0 and rc2=0 PASS 93 (11s) == sanityn test 94: signal vs CP callback race =========== 01:10:36 (1746162636) osc.lustre-OST0000-osc-ffff8800ae0f0000.idle_timeout=0 osc.lustre-OST0000-osc-ffff8800b60dc000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8800ae0f0000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8800b60dc000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0462903 s, 177 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 3.03324 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5764: 4702 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00768854 s, 533 kB/s osc.lustre-OST0000-osc-ffff8800ae0f0000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8800b60dc000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800ae0f0000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b60dc000.idle_timeout=debug PASS 94 (11s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 01:10:47 (1746162647) llite.lustre-ffff8800ae0f0000.fast_read=0 llite.lustre-ffff8800b60dc000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0206558 s, 102 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8800ae0f0000.fast_read=1 llite.lustre-ffff8800b60dc000.fast_read=1 PASS 95a (19s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 01:11:06 (1746162666) llite.lustre-ffff8800ae0f0000.fast_read=0 llite.lustre-ffff8800b60dc000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB) copied, 0.00246016 s, 5.0 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8800ae0f0000.fast_read=1 llite.lustre-ffff8800b60dc000.fast_read=1 PASS 95b (9s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 01:11:15 (1746162675) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (2s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 01:11:17 (1746162677) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00193479 s, 2.1 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (2s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 01:11:19 (1746162679) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0330975 s, 63.4 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (3s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 01:11:22 (1746162682) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0401185 s, 52.3 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (3s) == sanityn test 100e: DoM: read on open and file size ==== 01:11:25 (1746162685) 1+0 records in 1+0 records out 12000 bytes (12 kB) copied, 0.00378096 s, 3.2 MB/s PASS 100e (3s) == sanityn test 101a: Discard DoM data on unlink ========= 01:11:28 (1746162688) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.000702055 s, 5.8 MB/s PASS 101a (3s) == sanityn test 101b: Discard DoM data on rename ========= 01:11:31 (1746162691) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00162665 s, 2.5 MB/s PASS 101b (3s) == sanityn test 101c: Discard DoM data on close-unlink === 01:11:34 (1746162694) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00064715 s, 6.3 MB/s PASS 101c (4s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 01:11:38 (1746162698) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 00 02 00 00 00 e4 25 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000402:0x25e4:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.remote fh_bytes: 32 fh_type: 151 fh_data: 05 04 00 40 02 00 00 00 24 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000405:0x1c24:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.remote/subdir fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 00 02 00 00 00 e5 25 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000402:0x25e5:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.stripe fh_bytes: 32 fh_type: 151 fh_data: 05 04 00 40 02 00 00 00 25 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000405:0x1c25:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.stripe/subdir fh_bytes: 32 fh_type: 151 fh_data: 05 04 00 40 02 00 00 00 26 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000405:0x1c26:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! PASS 102 (3s) == sanityn test 103: Test size correctness with lockahead ========================================================== 01:11:42 (1746162702) striped dir -i1 -c2 -H crush /mnt/lustre/d103.sanityn fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1746162703 lockahead_test: lockahead_test.c:1191: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1746162705 Finishing test test23 at 1746162706 PASS 103 (8s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 01:11:49 (1746162709) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0111624 s, 91.7 kB/s oleg344-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg344-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps STAT a|m|ctime 1746162711 1746162711 1746162711 MDT a|m|ctime 1746162711 1746162711 1746162711 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00818046 s, 125 kB/s oleg344-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg344-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps STAT a|m|ctime 1746162711 1746162713 1746162713 MDT a|m|ctime 1746162711 1746162713 1746162713 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0199424 s, 51.3 kB/s oleg344-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg344-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps STAT a|m|ctime 1746162711 1746162718 1746162718 MDT a|m|ctime 1746162711 1746162718 1746162718 oleg344-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg344-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps oleg344-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg344-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps STAT a|m|ctime 1746162711 1746162718 1746162718 MDT a|m|ctime 1746162711 1746162718 1746162718 oleg344-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg344-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps STAT a|m|ctime 1746162711 1746162718 1746162727 MDT a|m|ctime 1746162711 1746162718 1746162727 PASS 104 (21s) == sanityn test 105: Glimpse and lock cancel race ======== 01:12:10 (1746162730) striped dir -i1 -c2 -H crush2 /mnt/lustre/d105.sanityn fail_loc=0x416 PASS 105 (28s) == sanityn test 106a: Verify the btime via statx() ======= 01:12:38 (1746162758) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (2s) == sanityn test 106b: Glimpse RPCs test for statx ======== 01:12:40 (1746162760) SKIP: sanityn test_106b statx() only test SKIP 106b (1s) == sanityn test 106c: Verify statx attributes mask ======= 01:12:41 (1746162761) SKIP: sanityn test_106c statx() only test SKIP 106c (2s) == sanityn test 107a: Basic grouplock conflict =========== 01:12:43 (1746162763) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.129912 s, 80.7 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5722 0x165a 0x280000401 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7670 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7670 First grouplock blocks second one PASS 107a (5s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 01:12:48 (1746162768) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.111301 s, 94.2 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5684 0x1634 0x2c0000401 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7670 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (9s) == sanityn test 108a: lseek: parallel updates ============ 01:12:57 (1746162777) 1+0 records in 1+0 records out 10000 bytes (10 kB) copied, 0.00387675 s, 2.6 MB/s fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB) copied, 4.27124 s, 2.0 MB/s PASS 108a (8s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 01:13:05 (1746162785) 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 3 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 5 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 6 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 7 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 9 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 10 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 17 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 19 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 20 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 22 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 23 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 24 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 25 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 26 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 32 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 33 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 34 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 36 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 37 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 38 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 45 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 47 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 48 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory oleg344-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 PASS 109 (579s) == sanityn test 110: do not grant another lock on resend ========================================================== 01:22:44 (1746163364) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: '/mnt/lustre/f110.sanityn' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934275 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-02 01:22:45.000000000 -0400 Modify: 2025-05-02 01:22:45.000000000 -0400 Change: 2025-05-02 01:22:45.000000000 -0400 Birth: - fail_loc=0 fail_val=0 Connected clients: oleg344-client.virtnet oleg344-client.virtnet PASS 110 (65s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 01:23:49 (1746163429) fail_loc=0x8000018a /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6553: 7460 Terminated $MULTIOP $DIR2/$tdir/mdt0dir/foodir/file2 Ow4096_c PASS 111 (7s) == sanityn test 112: update max-inherit in default LMV === 01:23:56 (1746163436) lmv.lustre-clilmv-ffff8800a818d800.qos_threshold_rr=90 lmv.lustre-clilmv-ffff88012aacd000.qos_threshold_rr=90 PASS 112 (6s) == sanityn test 113: check servers of specified fs ======= 01:24:03 (1746163443) lustre-OST0000-osc-ffff88012aacd000 active. lustre-OST0001-osc-ffff88012aacd000 active. lustre-OST0000-osc-ffff88012aacd000 active. lustre-OST0001-osc-ffff88012aacd000 active. MGC192.168.203.144@tcp active. PASS 113 (3s) == sanityn test 114: implicit default LMV inherit ======== 01:24:06 (1746163446) /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 PASS 114 (10s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 01:24:16 (1746163456) total: 3000 create in 7.92 seconds: 378.55 ops/second oleg344-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg344-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps oleg344-server: htree_dump_int_block: Ext2 directory block not found while calculating checksum for htree root oleg344-server: USE: f110 f911 fail_loc=0x80002401 fail_val=5 mkdir: cannot create directory '/mnt/lustre/d115.sanityn/f911': File exists PASS 115 (18s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 01:24:34 (1746163474) llite.lustre-ffff8800a818d800.intent_mkdir=1 llite.lustre-ffff88012aacd000.intent_mkdir=1 MD layout /mnt/lustre/d116.sanityn: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000bd0:0x3d8:0x0] 1 [0x240000404:0x3ba:0x0] mkdir /mnt/lustre/d116.sanityn/tdir0 setdirstripe -D -i1 /mnt/lustre2/d116.sanityn/tdir0 mkdir /mnt/lustre/d116.sanityn/tdir0/tdir11 lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none mdtidx FID[seq:oid:ver] lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none mdtidx FID[seq:oid:ver] PASS 116 (4s) == sanityn test 200: service remains healthy while able to process request ========================================================== 01:24:38 (1746163478) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=14 striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d200.sanityn fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.0551 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.0658 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.0816 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.0779 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.1319 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.0989 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.0884 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied4096 bytes (4.1 kB) copied, 20.0878 s, 0.2 kB/s , 20.0811 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.1136 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.1208 s, 0.1 kB/s 4096 bytes (4.1 kB) copied, 40.1046 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.1377 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.1216 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.1258 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.0991 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.1052 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.1146 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.1075 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.1008 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.1643 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.1487 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.1474 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.133 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.1631 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.1754 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.1485 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 80.2008 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg344-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800a818d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a818d800.ost_server_uuid in IDLE state after 0 sec PASS 200 (89s) cleanup: ====================================================== == sanityn test complete, duration 7087 sec ============== 01:26:08 (1746163568) sanityn: FAIL: test_16j dio copy 1 w/bsize 1049600 failed: 1 === sanityn: start cleanup 01:26:08 (1746163568) === Stopping clients: oleg344-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg344-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 01:28:01 (1746163681) ===